Our customers require us to develop software that is trustworthy and secure. Privacy also demands attention. To ignore the privacy concerns of users is to invite blocked deployments, litigation, negative media coverage, and mistrust. The Quality Engineering (QE) Security team’s goal is to minimize security- and privacy-related defects in design, code, documentation, and to detect and eliminate these defects as early as possible in the software development life cycle (SDLC). Developers who most effectively address security threats and protect privacy earn users’ loyalties and distinguish themselves from their competitors.
The QE Security team is responsible for the security component of the QE process. The goal of the team is to ensure that all customer product development tasks, including code and backend infrastructure, are secured as part of the SDLC. We are one of the following three pillars of software quality:
This process includes conducting threat models, system architecture and infrastructure reviews, security code testing and reviews, API and web application security testing, infrastructure security testing, penetration testing, and adherence to secure development standards.
For every new product, the QE Security team collaborates with all teams to create a product security test strategy and review the strategy with all the teams together. The security test strategy covers the following security testing aspects:
During the design stage, we conduct threat modeling for the application. The threat modeling session focuses on sensitive data and data flows across various components. The session helps us identify potential security defects in the design stage so that we can address them early. Once the Development team starts coding, the QE Security team conducts a security code review of the source code using both automatic tools and a manual reviewing approach. After we deploy the application into the staging environment, the QE Security team conducts web application and API security testing. We prioritize identified defects based on their severity and impact. The QE Security team works with the Development team and QE team to make sure that any critical or high security defects are addressed before the application is released into the production environment.
The trend in the security field is to integrate with the development process as early as possible with minimum interruption of their operations. The QE Security team should be able to speak the same language as the Development team to get their buy-in about security. Since every Development team is using Agile development process at Rackspace, the QE Security team also adapted the Agile process for our operations. We use the same Jira system as all other teams to track all our tasks and stories so that they are aware of our progress. For any incoming security request, we create a Jira ticket and add the requester as the observer. For identified defects, we create Jira defects in the Development team’s Jira project directly so that they can pick up the story and fix them quickly. The QE Security team also has daily standup to check the progress of the team members' Jira stories. We run retrospective meetings every two weeks to identify room for improvement in our process.
Threat Modeling is a core element of the overall Security Life-cycle (SDL). As part of the design phase of the SDL, threat modeling allows software architects to identify and mitigate potential security issues early, when they are relatively easy and cost-effective to resolve. Therefore, it helps reduce the total cost of development.
The threat modeling session is a technical discussion of the design and deployment strategy used by the product. The QE Security team creates a data flow diagram as the discussion continues. Depending on the complexity and type of product, the session might cover the following topics:
Threat modeling is an interactive, iterative process that covers the general use cases for the product, starting from when a customer first uses it. The model follows the flow of data through the product’s system and the product’s interactions with external systems.
The following example shows a data flow diagram for one of our products:
If, at a future date, the product is changed, a short threat model update session is scheduled to review the existing threat model and update it as needed.
The security code review is the process of auditing an application’s source code to verify that the proper security controls are in place to eliminate possible application security defects, and it ensures that developers are following secure development guidelines.
We use a commercial solution with an automatic security code-scanning tool that can scan thousands of lines of code in a short time. However, the findings generated by the tool might contain false positives. The QE Security team works with the Development team to verify each finding to filter out false positives. In addition, the QE Security team also conducts a manual security code review of key components of the source code. During the security code review, we focus on identifying the following common security defects as defined in the OWASP Top 10 Website:
We have code review checklists for different types of applications. The QE Security team leverages these checklists to make sure that the review is compreshensive. For example, the following list shows the authentication section of a security source code review checklist for a web application:
API security testing is a process of comparing the state of an API against a set of security requirements. Unlike a security code review, API security testing needs a fully functional application to run various security tests. Before they actively test the product’s systems, the QE Security team works with the teams to determine the appropriate scope for the API test. After the scope is determined, the QE Security team works with the Product team to determine a workable testing window for the testing to take place. At the scheduled time, the QE Security team begins the test with activities such as reading the API documentation (to understand how to use the API), creating test API calls, running security checks, and so on.
API security testing can be done using automatic tools or manually. Normally, it is a mix of manual verification and use of automatic tools. For example, we use Burp, an intercepting proxy, to view and modify the HTTP communication between the tester and the API. You might include certain test strings in the payload to verify whether the request body is subject to SQL injection by checking the response from the server. At the same time, you can also use intruder module from Burp to automatically send thousands of different test payloads to the server. We use the following checklist with different categories of tests that we should perform to provide comprehensive coverage:
Each category of security tests has a separate list of security tests. For example, the following security tests belong to the data validation testing phase:
Infrastructure security testing is the process of identifying, classifying, and prioritizing vulnerabilities in computer systems and network infrastructures. The process identifies threats and the risks they pose. We normally use automated testing tools, such as Nessus or Nexpose scanner, whose results are listed in the test report. For each vulnerability reported by the commercial scanner, the quality engineering security team manually verifies the reported vulnerability to remove any false positives from the final report of test findings.
Infrastructure security testing can identify the following security defects for computer systems and networks:
The average ratio between the number of Development team members and the number of QE Security team members is close to 50:1 in the industry. If the QE Security team is using any manual testing processes, they struggle to meet the security requests from the Development team, even if they also use automated tools. At the same time, the Development team releases their products faster and faster by leveraging technologies such as Continuous Integration and Continuous Delivery (CI/CD). Traditional security teams cannot handle CI/CD responsibilities because their testing last days or weeks.
To meet these new challenges, our QE Security team focuses on automation for both tooling and processes. Every member of this team is a software developer who can write code and work on automation. The QE Security team provides security tools, with a run time of minutes or seconds, that can be integrated into the development process. Without the tools, the Development team would move forward without security. At the same time, the QE Security team separately runs the same tools for better coverage with a longer run time. For example, both the Development team and the QE Security team use an automatic static code scanning tool. The Development team configures the tool to scan changed files for each pull request with a quick turnaround time. The QE Security team configures the tool to scan the whole source code repository every night by pulling the latest code for better coverage with longer scanning time. Together, we increase the tool coverage making both teams happy.
The QE Security team has been worked on various automation tools and has released the following open source tools. We are look forward to releasing more in the future.
Security is very important for the success of Rackspace. The QE organization includes security (and therefore the QE Security team) as one component of its overall responsibility. Other QE components are functional and performance testing. The focus of the QE Security team is similar to that of the Development team - build a product that is functional, secured, and performs as expected. The QE Security team works with other teams to identify and address security defects before the product is released to customers. Security testing is conducted throughout the entire SDLC so that security defects may be addressed in a timely manner. Together, we offer a fanatical security experience to our customers.
Use the Feedback tab to make any comments or ask questions.