How we captured AJAX requests from a website tab with a Chrome extension

Background

I assumed there would be a lot of Chrome extensions to monitor HTTP requests for AJAX calls, but the few we found such as the Chrome Postman Intercept extension seem to capture only the request, not the response. Even if Chrome DevTools has the network tab, it’s difficult to share these captured HTTP traces with teammates.

Our engineers divide the responsibilities between the front-end and the back-end, so we have to constantly copy / paste the HTTP headers and the JSON payload of DevTools in emails or Slack when debugging API problems . Sending multi-megabyte * .har files via Slack is just as tedious. Besides not being easy to share, inspecting HTTP traffic in Chrome DevTools is not ideal as there is no easy way to filter API calls or inspect JSON payloads. The network tab is designed for traditional HTML websites rather than modern single-page applications and APIs that return JSON.

Because of this waste of time, we decided to create a Chrome extension to make it easier to capture and debug these AJAX requests (and responses) from any website. The extension is designed for REST APIs, such as those that power single-page applications. However, monkey activity was required to perform the capture which bypassed the Chrome extension APIs due to the limitations of the Chrome APIs.

Issues with WebRequest approach

The Chrome WebRequest API is part of the Chrome Extension API set.

Initially, I looked at Chrome’s WebRequest APIs to capture API calls from an open browser tab. We have already used these APIs for our CORS and Origin Changer. It seemed like the most natural place to capture data.

However, we encountered a bug in which the WebRequest API does not expose an interface to read the body of the response. See the problem here on Chrome.

Although we were surprised by the lack of response to the body, we now understand a possible reason why most of the other extensions do not capture the responses. Our requirement was to capture the answer so that it could be shared with team members. Without answers, debugging would be painful and problem solving would be slow.

XmlHttpRequest event listeners approach

Since XmlHttpRequest  is used by all browsers for AJAX calls, I leveraged this to monitor the API calls through monkey patching.

There are several XmlHttpRequest listeners for hooking into:

• loadstart
• progress
• abort
• error
• load
• timeout
• loadend
• readystatechange

From the listener callbacks, I can get the event target, which is the XmlHttpRequest object itself. Looking at the XmlHttpRequest api documentation, I realized it has exact the opposite problem as WebRequest API. The object has interfaces to get the HTTP response but not the original request. There is a setRequestHeaders(), but no getter method to pair with it to get the request HTTP headers. In fact, it doesn’t even have a public method or property to get the original URL or path.

XmlHttpRequest Monkey patch.

To truly monitor all the data (i.e. request headers/body and response headers/body), I’d have to monkey patch it. The monkey patch allows us to log the data.

The key methods to patch are

• open() This method is called whenever a new AJAX call is initiated which I can then capture the method and url.
• setRequestHeader() This method is called when the request header is set. I can patch it so that every time the method is called, I save the headers as hash.
• send() This method is used to send data as part of the request. I can capture the request body from this patch.

Now, I can simply add addEventListener('load', event) to capture the response data.

There are a few resources on the web such as this article explaining how to monkey patch.

Execution Environment

With code ready for monkey patching, we have to execute it. Chrome Extension Tabs API has a method to execute the code: executeScript(). However, the code executed by executeScript() is executed in a different context than the code on the webpage, which is where the AJAX calls are made. Meaning the monkey patch won’t have any effect on the open website.

The solution: Injected script

Due to the isolated context, I had to take a different approach to successfully monkey patch the website. I had to use chrome.tabs.executeScript() to create a <script> tag injected on the website which would load the monkey patch code inside the website’s context.

  const actualCode = `
  var s = document.createElement('script');
  s.src = ${resourceUrl};
  s.onload = function() {
    this.remove();
  };
  (document.head || document.documentElement).appendChild(s);
`;

  chrome.tabs.executeScript(tabId, {code: actualCode, runAt: 'document_end'}, cb);

There are several ways to add the code into the script tag. Since we also wanted to add some nice UI elements, we decided to put the monkey patch code and the UI related code into a script file, which the resourceUrl links to. If you do this, be sure to add the resource url to the web_accessible_resource in the manifest. More information on Web Accessible Resources