Earlier this month, PerimeterX co-hosted a Tweet Chat with IT Security Guru on the topic of Shadow Code and invited a variety of industry experts including analysts, influencers and executives to weigh in on this little-known threat. The conversation lasted for an hour and delved into the issue from the perspective of DevOps, IT security, e-commerce and beyond. Participants included the following individuals:
- Ameet Naik, Cybersecurity Evangelist, PerimeterX
- Carlos Kizzee, EVP, Intelligence Operations and Legal Affairs, RH-ISAC
- Jamie O'Meara, Global Solutions Architect, Snyk
- Kim DeCarlis, CMO and Security Expert, PerimeterX
- Quentyn Taylor, Director of Information Security at Canon for Europe, Middle East and Africa
- Richard Stiennon, Chief Research Analyst, IT Harvest
- (Moderator) Tony Morbin, Editor-in-Chief, IT Security Guru
- (Moderator) Yvonne Eskenzi, IT Security Guru
Q1: Have you heard the term #ShadowCode before? If yes, what do you understand it to mean?
Carlos: I think of #ShadowCode as the generally overlooked and often unknown third-party or “nested service provider” code that is incorporated into your e-commerce websites without the knowledge of the security team or awareness of its impacts on security, latency or compliance.
Jamie: #ShadowCode is the use of third-party scripts and libraries in a web application. 80% of code used in applications today originates outside an organization. External code, called open-source, provides accelerated value delivery, it also represents a risk to the organization.
Quentyn: #ShadowCode is code that's been cut and pasted from other third-party locations and may not have been vetted to the same degree as own written code. It doesn’t mean it’s inherently insecure though.
Ameet: Application development today makes extensive use of third-party scripts and open source libraries, which are great for innovation and agility, but the end result is you don't really know what code is running in your application. This, in essence, is the #ShadowCode problem.
Richard: To be honest, no! But I had recently encountered the issue when a virtual conference partner questioned the security of embedding script in their platform. What's the issue? You can read the script to verify what it does!
Q2: From a business and marketing perspective, why should companies be aware of the code that they host on their digital ecosystem?
Richard: Lots of reasons. You may need to know the whole ecosystem to debug or resolve issues. If you need to comply with GDPR you need to know who is processing PII of data subjects and be able to audit them.
Carlos: For the same reason that a company needs to be aware of who is behind their retail counter accepting payment in a brick and mortar store. Code in digital transactions can leverage access to customer confidential information and exfiltrate that information—customer cardholder sensitive information—to parties beyond the intended recipients.
Ameet: Data privacy regulations are only increasing in scope and severity and the penalties are harsh.
Jamie: Websites represent systems of engagement in most organizations. They attract the buyer to transaction and provide a quick and enjoyable user experience if done correctly. Trust is crucial; less trust then less business.
Ameet: Companies should care about keeping their customers safe - online or offline. #ShadowCode provides an entry point for cybercriminals.
Kim: Brand reputation can easily be diminished due to a data breach or compliance issue. Website owners MUST work with Security and IT teams to ensure that their code is reviewed and implement tools to minimize exposure.
Quentyn: You are responsible for all lines of code that get executed on your customers’ PCs and that includes website code—think Magecart. There was also the case where malware was in the C library that companies were using neatly bypassing the normal supply chain issues.
Q3: What are the consequences of not being aware of #ShadowCode from a security perspective?
Kim: While using third-party code can help with agility, it introduces cyber risk. #ShadowCode that has a vulnerability can be modified to send PII to unauthorized sites for nefarious purposes. Recent Magecart incidents are a result of this and led to brand damage and fines.
Carlos: You don’t know the extent of who has access to sensitive financial transactional data in your commercial transactions because you don’t know the extent of the code executing on those pages. This means that you also don’t know where the transactional data is going and what other parties may be doing with the sensitive information associated with the digital transaction.
Quentyn: Well, the main issue is that #ShadowCode can be seen as a supply chain issue—supply of code that bypasses normal checking.
Ameet: The digital supply chain is a real challenge to verify and secure. It's not only the third-party code that you use, but also all the external code that your suppliers use!
Quentyn: And the code your suppliers of your suppliers use. What was that case with the compromised visual C libraries and the remote control software? The victims weren’t even on the target list.
Ameet: You cannot secure what you cannot see. Client-side #ShadowCode is a real blind side for InfoSec teams.
Quentyn: I would hope that many are performing source code review. What’s more worrying is when you depend on compiled binaries and have no access and have to trust.
Ameet: Especially when that code runs directly on your customers' browser without you ever seeing it!
Richard: We have seen this play out with malicious code being delivered to customers via unverified ads.
Q4: How is #ShadowCode impacting different job functions such as #CISOs, #infosec teams and #DevOps teams?
Carlos: This unknown and unmonitored code is not always the result of a nefarious threat actor. It can also be the result of a developer on a tight schedule who leverages code from a third-party code repository without knowledge of all aspects of the code and its impacts on the network.
Quentyn: For CISOs it’s both a risk and an opportunity. #ShadowCode probably affects DevOps more as it's a huge credibility issue for them. How much of this did you write yourself? How much of this code did you understand before ctrl+v?
Kim: Let's not forget about marketing and CMOs. They steward brand reputation and a negative #ShadowCode incident has huge implications here. This also means that #marketing teams, chief digital officers and CMOs need to be planful when they add third-party scripts to their websites or web apps. Make friends with your CISO and do the security check to avoid downstream damage!
Carlos: Marketing and CMOs need to be a part of the #ShadowCode solution. Their interests are impacted by the effects of code executing in e-commerce transactions which can cause latency and lead to customer abandonment.
Kim: 100% agree. The need for cross functional collaboration is HUGE! CMO + CISO = reduced risk from #ShadowCode gone awry.
Jamie: DevOps, now DevSecOps, has expanded to include security and shift the landscape left. Automation in the CI/CD world is everything. Source Code platforms, I just coined that term in this tweet, are here to offer everything for your source code. One area they can help with is deployment and testing using tools to identify open source and #ShadowCode scripts.
Ameet: #ShadowCode enables agility for #devops teams, but much like we saw with Shadow IT, CISOs are still ultimately accountable for data privacy and risk.
Quentyn: It enables agility if they understand what they copy. If not, it’s a risk. Also note that dependent on where the code is obtained from, there may also be licensing risk.
Q5: Can you provide any examples of when #ShadowCode has had a negative effect on a company in the past?
Ameet: The string of Magecart attacks on major brands recently have been a direct consequence of #ShadowCode. Most of these have resulted in the theft of credit card data, but some incidents have also scraped usernames and passwords leading to account takeover attacks.
Kim: Unfortunately there are many! Delta Airlines suffered an attack as a result of #ShadowCode. So did Best Buy. No one likes sharing PII with a company that might put it at risk. And no marketer wants to deal with the bad press that results.
Richard: Pretty much anytime you update/install WordPress, Joomla, Drupal, or any plugin, you are exposing yourself to somebody else's code.
Q6: What are the implications for specific industries such as #ecommerce, travel and #elearning?
Carlos: In addition to the obvious e-commerce security implications (loss of cardholder data on the merchant side), there are also impacts to merchant revenue like the loss of efficiency in the transaction due to latency from code operations that could translate into lost sales due to customer abandonment based on the resulting latency.
There are also concerns with the impact of code that captures customer identity and interests from the transaction and uses them to steer the customer to competitor products.
Kim: I worry a lot about e-learning. They are often not as mature in their security practices as other industries like financial services and #healthcare. And they have PII from students and others where there are additional rules and implications. They must protect their web apps from bad #ShadowCode.
Any industry that uses #ShadowCode to speed their time to market with new features is at risk. Since e-commerce involves username, password info and often credit cards, they are a big attack target. Threat actors reuse credentials to empty accounts or sell them on the dark web.
As a result ANY industry that collects this type of info needs to take precautions. Gain visibility on your #ShadowCode and continually monitor it for changes.
Ameet: The travel industry has been a big target for #Magecart gangs with high profile attacks on major global airlines. E-learning platforms are also often used by kids taking classes from home. Protecting their data privacy is not only required by the law in many jurisdictions, but it's also the right thing to do!
Q7: What is the best way for security teams to mitigate the effects of #ShadowCode?
Carlos: Review code to be used to identify potential vulnerabilities. Keep an eye on logs from your runtime environment and review them for anomalous activities related to code execution. From a compliance standpoint, set and enforce clear policies for service providers and internal dev teams on the use of third-party code and ensure that it clearly requires all third-party code to be validated by evaluation and verification tools.
Ameet: Continuous visibility and monitoring. Third-party code goes directly from the vendor to your customers' browser - you don't know what's in it. Static analysis only provides a point-in-time snapshot.
Jamie: Mitigation starts early, as part of the software development lifecycle. Automation and tools will be the first stop. We love our tools, and I've seen a lot of incredible innovations around security for developers.
I think process change can also help a little. Using paired programming, two programmers work together to deliver new features, HA, resiliency, and security.
Richard: Start with a survey of all your external code. Scan your web assets for such code. Verify that it 1. should be there, and 2. does only what it is supposed to do. Next step: do that continuously and work it into your devsecops process.
Q8: Some say content security policies (CSPs) are sufficient to address #ShadowCode. Is this accurate?
Carlos: They are an “arrow in the quiver”, but CSPs are by no means the “magic bullet.” They can mitigate common scripting and code injection attacks, but they are but one control. Remember DEFENSE IN DEPTH…patch, disable unnecessary ports/services, perform code review, carefully review third-party service providers…
Quentyn: Should you use a CSP? Yes. Will it protect you from #ShadowCode? It depends. It’s part of a possible solution. A CSP can help if the #ShadowCode reaches out to a third-party location for resources. But if the code runs on the domain that the CSP allows then it will not be relevant.
Ameet: CSP is a useful tool in the fight against malicious code injection. However as you point out, it's not a set-and-forget solution.
Quentyn: Indeed, and it stops a dev from putting in code that calls third-party locations without control, but it isn’t a panacea.
Richard: Not in my opinion. A CSP, if deployed properly, helps ensure authorship and source of code, but not what the code actually does.
Q9: #ShadowCode has been responsible for many recent #Magecart attacks that incur GDPR and CCPA penalties. How much progress have we made towards achieving compliance with data privacy regulations?
Carlos: Compliance and the threat of penalties gets the attention of the Chief of Compliance/Chief of Legal and that helps. But progress to eliminate this problem lies with the marketing team, the dev team and the CISO’s security operations team. We can make a greater stride towards addressing this problem by aligning all aspects of the problem and working together for an environment where companies are committed to the environment that can assure safe, secure, efficient and compliant code.
Richard: We are only at the beginnings of being able to comply with GDPR, CCPA, etc. Most orgs take a wait and see approach to regulatory enforcement. GDPR for instance calls for "state of the art" security. We don't even know what that is!
Kim: Most organizations have a plan and have #CheckedTheBox on #compliance. But are they really compliant? Certainly big brands like @British_Airways @Macys @claires thought they were compliant, but they still had a data breach and many paid huge fines. #ShadowCode runs on the client side which can be the #blindside for many organizations. Need to have tools and processes to guard against this.
Quentyn: Sadly, I don’t believe a lot...most breaches are still an article 32 failure.
Ameet: According to a recent survey from Osterman Research, only 30% of organizations reported their websites were fully compliant with data privacy regulations.
Q10: Where do you see the issue of #ShadowCode in 5-10 years?
Carlos: If it is easier to deploy, human nature and time pressures will ensure the continued reliance on third-party code without due diligence. Criminals being criminals, will go where there is money. I see these two trends providing continued opportunity for criminals to plant seemingly innocent code in public code libraries in the hopes that a well-meaning developer or service provider uses it in a customer project. We need to address all aspects of the problem.
Jamie: Application frameworks have a short timespan. We love change in the application development space, as cool or more performant solutions evolve. Combine this with new products and marketing techniques, and #ShadowCode will continue to grow over the next 5-10 years.
Quentyn: #ShadowCode will be an issue that will get far worse before it gets better—the implications to products that can't be updated are concerning.
Kim: Wish I had a crystal ball. #ShadowCode will certainly be better understood in the next few years. Could the pendulum swing to not allow any third-party code use? Doubtful! Need for speed and agility in web app development will supersede.
Time is now for organizations to understand the #ShadowCode risk, and build plans. Then stay on top of the evolving techniques to protect against malicious activities.
Quentyn: I think that signing of known good components will be more common so that people can have more (but not total) trust in what they run. I think it's an issue that's hidden at the moment from most.
Richard: Signing code is a great start. At the very least it increases the cost for the attacker.
Ameet: #ShadowCode is here to stay. We will learn to live with it. Continuous visibility solutions and more client-side policy controls will help us regain confidence in our web applications.
Richard: Growing exponentially until our AI overlords step in to straighten it all out. :-)
Ameet: Haha. AI and ML techniques are very helpful in analyzing runtime behavior and separating out the good from the bad.
We’d like to extend our thanks to everyone who participated in the Tweet Chat and hope industry professionals continue this discourse. We look forward to hosting more events on this topic.
For more on Shadow Code, view the variety of PerimeterX resources available on the topic here.