/var/log/DMIT-NOC.log

There is no further updates. Everything is work as expected.

DMIT’s service is unaffected for now.

Cogent’s capacity is impact for 3 hrs on last 24 hrs. We observed the congestion has been relieved for now.

Due to the Pacific Palisades wildfire in Los Angeles; there are many place has experience power outage include DTLA.

The following place reported power quality event and power outage in recent hour.

CoreSite LA1 (One Wilshire Building)

Digital Realty LOS1 (600 W 7TH)

Both site has report the they sucessed switch power source to generator or backup.

Please be prepared any possiable service downgrade.

DMIT will keep you posted.

The first batch of barebone are delivered; but the CPU and Memory are waiting for shippment.

Out of Stock Notice of DMIT Los Angeles Data Center

DMIT was unable to complete the expansion at the end of December 2024, due to HPE delivery overruns.
To ensure stable operation for existing customers, DMIT suspended sales of most instances in Los Angeles.

We expect to resume sales of all regular instances within 2 weeks.

Please contact us if you have any urgent need on specific one.

Network downgrade has been detected

Impact: CMIN2, Los Angeles.

The latency to China Mobile of LAX.EB, and LAX.Pro has increase.
No increaed packet loss rate has been observed.

LAX.Pro used CMIN2 for China Mobile inbound, the latency will also be impacted.
Notes: The only low latency inbound optimization for China Mobile is CMIN2.
CM-on-Net (CMI AS58453) has at least 150-180ms during normal time.

Issue has been reported to China Mobile International NOC already.
===
Jan 3, 2024 1AM EST
No update from CMI, but the latency return to normal hours ago.

Restored

Done

DMIT HKG Maintenance:

In 3 hours, upto 30 mins.

Replace / Upgrade Edge Router routing engine.

Observed CT CU routes has been withdrawn by CMI on Tokyo EB.

Waiting updates from CMI.

/var/log/DMIT-NOC.log pinned «From China Telecom, For Los Angeles CN2 GIA. ==== Dear customer, Please kindly be informed that there will be an planned maintenance and more information as below. Time window (Date/Time): 2025-01-10 16:00:00 - 2025-01-22 22:00:00 UTC 2025-01-11 00:00:00…»

From China Telecom,
For Los Angeles CN2 GIA.
====
Dear customer,

Please kindly be informed that there will be an planned maintenance and more information as below.

Time window (Date/Time):
2025-01-10 16:00:00 - 2025-01-22 22:00:00 UTC

2025-01-11 00:00:00 - 2025-01-23 06:00:00 UTC+8

Maintenance Description:
Facilities relocation

Maintenance Location:
American company, International

Service Impact:
The circuit will experience outage up to 360 minutes during the maintenance window.

CMI NOC indicated the issue has been resolved.

DMIT has confirmed this resolution after 3 days of observation.

Issue resolved;

Encounter JunOS software design defact.
VRF table refresh (deactivate & activate, table name remain unchange) make the VRF obtained a new FIB Table ID;

The PBR (which use "name" of table) does not update the Table ID after that.

Which bring the VM traffic been directed to a blackhole.

Resolved by disable PBR in Firewall, then reactivate.

Unexpected issue happend on TYO.Pro; investigating

We worked with CMIN2 to have the LAX.EB's current situation to be relieved.
Please check again.

CTGnoc report, partial Tokyo CN2 CTGnet also been detour because NCP failure.

DMIT already aware about LAX EB packet loss on China direction.

We observe there is no packet loss on PtP segment.

The CMIN2 NOC response there is an issue on NCP submarine. Their capacity has been impacted due to the submarine issue.

DMIT will keep you updated.

Ongoing

Edge Router hardware swap.

Time: 10AM-1PM JST, 12/18/2024

Expected maximum downtime: 60mins

Adjusting TYO network arch, some product experienced downtime, and it has been fixed now.
There are couple works need to be done before return to stable.

Observed TOR issue on TYO, dispatching engineer.
Done

- Extend 30 mins.
- Extend 45 mins again. Due to the initial RH does not record the label correct; it took many time to trace and re-cable.
- Ready in 11:30 JST

TYO:

Pending reboot action in an hour, expect VM reboot, and upto 1hr.

LAX:
Emergency Rebooting one of hypervisor; replace NICs.
Done.

/var/log/DMIT-NOC.log pinned «Schedule Maintenance: Location: Tokyo; Date: Nov 18 ~ Dec 18, 2024 Content: - Network Upgrade - Hardware Upgrade - Storage Upgrade Impact: - Network jitter, instability and downgrade; - Possible SSH Key change due to hardware migrate; - Experience downtime.»

Schedule Maintenance:
Location: Tokyo;
Date: Nov 18 ~ Dec 18, 2024

Content:
- Network Upgrade
- Hardware Upgrade
- Storage Upgrade

Impact:
- Network jitter, instability and downgrade;
- Possible SSH Key change due to hardware migrate;
- Experience downtime.

LAX:
Emergency Rebooting one of hypervisor

Done; upgrade the another RE;
Should expect no traffic stop.

Traffic switching in progress;
Routing Engine reboot in progress.

There is a pending upgrade on LAX's MX Edge Router.
Expect maximum 30 mins down time and 60 mins network downgrade.

Time: 3PM ~ 6PM PDT.

DMIT.io will also be impacted at the same time.

Interface configuration error. Revised.

As far as we see the link of CMIN2 returns to normal.

The maintenance is almost complete; we are verifying the status;

The CMIN2 issue happened near days is covered by this statement; we expect to experience network degradation sometime.

We will let you know once the status returns to normal.

Between Oct 26 ~ 31, we will move out the current network equipment and clean up the legacy things. We should expect no more issue but please wait for the further confirmation.

Replaced a new interface and observed it being good.

We will monitor for 24 hours.

The problem seems caused by dirt port on the MMR.

=======
Last reply from Coresite: “Patching was moved and cleaned to new LOA demarc. Light levels: rx from you +3.12 dBm and they are tx +7.34 dBm”

=======
Earlier messages from Coresite: “I traced down the issue and cleaned the ports and fiber ends. Can you please let me know if that fixed the problem?”

Coresite ident an issue on the cross connect; moving to a new interface, CMIN2 currently been disconnected.

193.41.250.181 is on CMIN2 routers;

193.41.250.250 is the Anycast Gateway IP which has ICMP response rate limitation.

Noticed the issue on LAX EB;

DMIT is now working with CMIN2 to figure that out.

The iLO/IPMI of DMIT.io cluster has some issue, working with on-site to fix it before put server online.

The on-site migration task takes longer than expected; We will keep you post here.

DMIT.io website and database migration pending.
expect to start on 12PM PDT.

LAX Pro Edge Router hot cut in progress.
Expected no impact but things happened.

Done.

Due to recent contractual and market changes, as well as uncertainties in the Hong Kong market that do not align with DMIT's operational philosophy, we have made the strategic decision to discontinue the HKG EB (excluding v2) series services.

Service Discontinuation

Effective November 20, 2024, your HKG EB (excluding v2) service will be discontinued. This decision has been made after careful commercial consideration and is final.

We kindly ask you to adjust your operations and back up your data before this date to avoid any unnecessary loss.

Refund and Compensation

For HKG EB (excluding v2) services with expiry dates beyond November 20, 2024, we will:

- Refund the residual value of your service.
- Credit your DMIT account with an additional 10% of the renewal price of your service.
- You may choose to receive the refund either to your DMIT account or via your original method of payment. DMIT will cover all payment processing fees.

Exclusive Discounts on Alternative Services

As part of our commitment to providing stable and cost-effective solutions, we are offering exclusive recurring discounts on the following product series:

- LAX.Pro & EB Series: 20% recurring discount
- TYO.Pro Series: 15% recurring discount
- HKG.Pro Series: 20% recurring discount
- HKG.EBv2 Series: 25% recurring discount
- HKG/SJC T1 Series (excluding WEE): 30% recurring discount

To avail these discounts, please submit a ticket request by December 31, 2024. You are entitled to one discount per series for each HKG EB (excluding v2) service you hold, allowing you to request discounts for each of the above series once.

Please Note:

- Discounts apply only to new service purchases and are not applicable to renewals.
- There is no limitation on the billing period for the new services.
- Discounts cannot be combined with other promotions.
- Eligibility is limited to customers with HKG EB (excluding v2) services expiring after October 10, 2024.

Special Product Offers

In addition to the above, you may request orders for the following special products:

- LAX.Pro.WEE
- LAX.Pro.MALIBU
- LAX.Pro.PalmSpring
- LAX.Pro.Irvine
- LAX.sPro.Fixed
- LAX.EB.WEE
- LAX.EB.CORONA
- LAX.EB.FONTANA
- HKG.Pro.VICTORIA
- HKG.Pro.LOKMACHAU
- HKG.EB.WEEv2
- TYO.Pro.Shinagawa

Submit your ticket to request by December 31, 2024. All orders must be on an annual payment cycle.

For each HKG EB (excluding v2) service you hold, you may request orders for any two of the above special products (they can be the same product).

Please Note:

- These special offers cannot be combined with other promotions.

- Eligibility is limited to customers holding HKG EB (excluding v2) services expiring after October 10, 2024.

Example of Compensation Plan Usage

For clarity, if you have a HKG EB TINY annual payment service, you can:
- Purchase a LAX.Pro TINY monthly payment service at a 20% discount.
- Purchase a HKG.EBv2 annual payment service at a 25% discount.
- Request orders for LAX.Pro.WEE and LAX.EB.WEE.

These options can be utilized simultaneously and are not mutually exclusive.

We sincerely appreciate your continued support of DMIT and apologize for any inconvenience this transition may cause. We are committed to providing you with superior service and look forward to serving your needs with our new offerings.

Sincerely,
DMIT Cloud Services.