A SWAT team for enterprise app woes

When an enterprise application doesn't perform as well as expected, finding the cause can be a daunting task. Is it the server? Or the database? Maybe it's the application itself, or some strange interaction between the two? The end user may blame the application itself, but the fault could also be on the network, say with a router or the firewall. Neither system administrators nor application management personnel have the complete view of the system needed to diagnose the problem.

Over the past year, the Coast Guard Operation Systems Center (OSC) has had a dedicated task force in place to troubleshoot complex problems with its own applications. And now it has begun offering the six-person team to other Coast Guard divisions as well, according to Lt. Anthony Baird, government project officer for the Performance Recording Operational Troubleshooting Evaluation and Consulting team.

The PROTEC unit can perform baseline testing of a new application before it goes into deployment to gauge how it will run under a full operational load. The team can also tease out where the problem resides for an application already in production use that is not operating as expected.

This is difficult work, approaching that of forensics. The applications being studied can run on anywhere from a server or two to up to 40, and the problems can take place almost anywhere the Coast Guard carries out operations, geographically speaking. The applications they have characterized run the gamut from Web servers, database hosting, e-mail financial applications, to specialized Coast Guard applications such as missile tracking, rescue support applications and apps that track ocean currents and tides.

"We can see how well an application is being utilized right here on OSC, or in Guam, Alaska, San Francisco or wherever," said Charles Asbury, a system engineer on the team. Performance is looked at, both from across the entire system, as well as evaluated from an end user's perspective.

The first step of diagnosing trouble is to understand where the problem actually resides. It could be a network issue, an Web server issue, an application issue or one involving the database. Once the problem is pinpointed, it can usually be fixed quite easily. Pinpointing the problem can take some effort, however.

To get a better handle of what is taking place, the team will look at network and server usage, and even watch what calls the program code itself is making. In some cases agents need to be installed on the same machine as the client software itself, as to watch the performance.

The team uses a variety of software to monitor performance. One application is Opnet Ace, from Opnet technologies. Opnet Ace is used to capture baseline statistics for how much bandwidth the system is using. It also comes in handy for troubleshooting.

These days more sophisticated tools are needed to understand what is going on with a network, said Russ Elsner, an associate vice president of Opnet. Opnet offers a range of what it calls application performance management to do the job of watching the traffic and filtering the results.

"Plugging a single packet-capture device up to your network and looking at a simple tracefile just doesn't work any longer," Elsner said. "There are so many moving parts in modern networks."

Other tools include Opnet's IT Guru, Hewlett-Packard's HP Loadrunner, HP Diagnostics and Simena's Network Emulator.

Even with such tools in place, it is still up to the team's engineers to figure out what is going on. "The tools provide us with clues where to look, but most of the knowledge is understanding the tools and where to go from what the information provides us," Asbury said.

In many cases problems will come about not so much as the result of one piece of technology failing, but rather because of interactions between how multiple products are configured.

In one case, the team was dispatched to diagnose a server that was taking up to two minutes to respond. It was one of 40 spread out across four different zones. The problem turned out not to be the server, though, but rather a firewall that would cut off connections after 30 minutes, in this case between the seemingly malfunctioning server and another server it was communicating with. The team reset the application to log in every 15 minutes, eliminating the problem, said Jason Kowalski, system engineer for the team.

In another case, the team confronted an application that performed adequately well in development but suffered performance problems when it was moved into production. Because the application pulled in hundreds of data fields it was difficult for the development team to determine where the bottleneck was. They guessed that the problem was with either the servers or the video cards, which they suspected couldn't keep up with the graphical output. In this case, however, the problem turned out to be three lines of ill-conceived code that were slowing the application. The Air Force has also used Opnet software to help diagnose application performance problems.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.