{"id":97908,"date":"2023-09-07T18:43:05","date_gmt":"2023-09-07T22:43:05","guid":{"rendered":"https:\/\/jumpcloud.com\/?p=97908"},"modified":"2023-11-01T12:21:49","modified_gmt":"2023-11-01T16:21:49","slug":"security-update-june-20-incident-details-and-remediation","status":"publish","type":"post","link":"https:\/\/jumpcloud.com\/blog\/security-update-june-20-incident-details-and-remediation","title":{"rendered":"[Security Update] June 20 Incident Details and Remediation"},"content":{"rendered":"\n
On 2023-07-12, we alerted customers to a security incident that occurred starting on 2023-06-20. Now that our investigation has concluded, we want to share some additional information around what happened, what we\u2019ve learned, and what our plans are to continuously improve our security posture for the future.<\/p>\n\n\n\n
We would like to thank our customers for their patience and understanding for the mandatory key rotation on 2023-07-05. We want to also thank our customers and our community for your patience while we investigated and remediated this incident. We aim for full transparency and disclosure, while maintaining the integrity of the investigation.<\/p>\n\n\n\n
2023-06-20 a sophisticated North Korean threat actor successfully spear-phished a 探花大神 software engineer, causing them to download malicious code to their 探花大神-issued device, which gave the threat actor developer-level access to 探花大神 environments.<\/p>\n\n\n\n
2023-06-22 the threat actor used developer-level access gained on the engineer\u2019s endpoint to pivot to other 探花大神 systems. From there, they were able to launch workloads to run at a later date in our container orchestration system.<\/p>\n\n\n\n
2023-06-23 02:21 探花大神 security tools alerted on anomalous activity taking place related to the compromised employee account. System access was revoked and known affected credentials were rotated. A number of mitigating actions that are detailed below were initiated at this time.<\/p>\n\n\n\n
2023-06-27 15:13 探花大神 Security noticed that developer access was used to run a workload that activated in our container orchestration system. We did not see evidence of customer impact at that time. <\/p>\n\n\n\n
Containment, eradication, and remediation efforts continued at this time. Credentials were rotated, infrastructure was rebuilt, code deployment was frozen, and a number of other actions were taken to further secure our network and perimeter. Additionally, our prepared Incident Response (IR) plan was activated. Our IR partner was engaged to analyze all systems and logs for potential activity. As part of our IR plan, we contacted and engaged law enforcement in our investigation. Forensics and investigation work continued to ensure the full scope and impact of the malicious activity was understood.<\/p>\n\n\n\n
2023-07-04 探花大神 identified and rebuilt the last impacted system. No further indicators of compromise have presented themselves on 探花大神 systems since this date.<\/p>\n\n\n\n
2023-07-05 探花大神 discovered an anomaly in our database records ultimately identifying the intent and impact of the attack. We discovered database injection that occurred on the 27th to instruct target devices to download malware. This occurred on fewer than 10 devices total across fewer than 5 organizations. We immediately contacted the organizations to notify them of the impact and ensure that there was no further exposure to them. We also took the proactive measure to force rotate all API keys once we had evidence of customer impact.<\/p>\n\n\n\n
Our audit of the entire database through objective analysis of this anomaly leads us to have extremely high confidence that we know the exact impact of the incident and have a comprehensive list of impacted devices.<\/p>\n\n\n\n
Further investigation, containment, and remediation actions continued to secure our environment. These actions are detailed below.<\/p>\n\n\n\n
2023-07-12 探花大神 published a public statement<\/a> advising the public of a security incident that took place and was the reason for the mandatory API key rotation. 探花大神 Security detected the compromise and responded accordingly to investigate, contain, and remediate the attack.<\/p>\n\n\n\n Indicators of compromise were thoroughly investigated, and a variety of containment and remediation strategies have been employed to secure the 探花大神 environment.<\/p>\n\n\n\n Upon discovery of compromise and an active phishing campaign of engineering and development employees, access to the 探花大神 application infrastructure was revoked for a large portion of users and roles. This was implemented to prevent further exposure by other potentially compromised employee endpoints until full scoping of incident impact could be completed.<\/p>\n\n\n\n All IAM permissions were reviewed, rearchitected, and restored based on necessity and job function relevant to various roles and users. The hardening of users and roles will limit the potential impact of compromised accounts and endpoints. Elevated access now requires manual authorization by multiple parties to prevent unwanted privilege escalation. Robust monitoring and alerting provides visibility to review and audit elevated access activity.<\/p>\n\n\n\n Upon discovery of anomalous activity, action was taken to rotate all 探花大神 customer API keys. Although no exposure was found at that time, we acted to secure customer environments from the possibility of compromised API keys. A notification was sent to customers informing them of the forced rotation with instructions to resume normal operation in their environment. An article was also published on the 探花大神 support site with the same contents as the email sent to customers.<\/p>\n\n\n\n During the investigation, all infrastructure affected by the threat actor was identified and completely rebuilt from scratch to further ensure that all persistence mechanisms of the threat actor were removed. Every credential and key in the 探花大神 environment was rotated to ensure no lingering access existed for the threat actor to take advantage of.<\/p>\n\n\n\n To prevent any potentially compromised source code from being deployed into the production environment, a deployment freeze was implemented early in the incident. We verified that no source code or binary releases were compromised in this incident.<\/p>\n\n\n\n All 探花大神 internal users and administrators were forced to rotate their credentials. All user endpoints were audited to verify that security tools are present and functioning correctly. No evidence of further employee compromise was found.<\/p>\n\n\n\n A number of measures were taken to expand monitoring capabilities. Indicators of compromise were added to security tools as they became available. Routine monitoring and system checks were expanded to include new indicators in order to provide visibility into new attempts by the threat actor. Monitoring of 探花大神 employee user access and API key activity improved visibility to ensure all activity is expected and appropriate for business processes.<\/p>\n\n\n\n 探花大神 engaged third-party incident response services to assist in the investigation, containment, and remediation of the incident. Forensics was conducted in all affected environments, and detailed reports were provided to contain and remediate the impacted systems. Contact with the appropriate law enforcement agencies has been established. <\/p>\n\n\n\n The investigation revealed that the threat actor injected agent commands to run on fewer than 10 devices across fewer than 5 total organizations. At that point 探花大神 promptly contacted the affected customers to inform them of the attack and offer assistance.<\/p>\n\n\n\n At appropriate milestones, 探花大神 provided communications to customers and the general public. This was accomplished through customer email notifications, 探花大神 support site articles, as well as interactions between customers and 探花大神 support staff. Internal communication channels were created to bridge the gap between customer inquiries and various members of security and engineering teams.<\/p>\n\n\n\n Fewer than 5 organizations and fewer than 10 total devices were successfully targeted by the threat actor. 探花大神 made contact with all affected customers prior to public announcement. If your organization was not contacted and informed of impact, it was not impacted by this incident.<\/p>\n\n\n\nHow do we know this attack vector is closed?<\/h2>\n\n\n\n
Access Revocation and Granular Restoration<\/h3>\n\n\n\n
API Key Rotation<\/h3>\n\n\n\n
Infrastructure Destruction and Rebuilding<\/h3>\n\n\n\n
Source Code and Binary Validation<\/h3>\n\n\n\n
User Credential Rotation and Endpoint Verification<\/h3>\n\n\n\n
Enhanced Monitoring<\/h3>\n\n\n\n
Third-party Incident Response Services and Law Enforcement<\/h3>\n\n\n\n
Communication and support for customers<\/h2>\n\n\n\n
How do I know if I was impacted?<\/h2>\n\n\n\n
What can I do?<\/h2>\n\n\n\n