[{"data":1,"prerenderedAt":708},["ShallowReactive",2],{"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change/":3,"navigation-en-us":36,"banner-en-us":453,"footer-en-us":469,"Igor Wiedler":680,"next-steps-en-us":693},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"This SRE's HAProxy Config Change: An Unexpected Journey","This post is about a wild discovery made while investigating strange behavior from HAProxy. We dive into the pathology, describe how we found it, and share some investigative techniques used along the way.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681844/Blog/Hero%20Images/infra-proxy-protocol-wireshark-header.png","https://about.gitlab.com/blog/this-sre-attempted-to-roll-out-an-haproxy-change","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next... \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Igor Wiedler\"}],\n        \"datePublished\": \"2021-01-14\",\n      }",{"title":17,"description":10,"authors":18,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...",[19],"Igor Wiedler","2021-01-14","This blog post was originally published on the GitLab Unfiltered\nblog. It was reviewed and republished on\n2021-02-12.\n\n{: .note .alert-info .text-center}\n\n\n## TL;DR\n\n\n- HAProxy has a `server-state-file` directive that persists some of its\nstate across restarts.\n\n- This state file contains the port of each backend server.\n\n- If an `haproxy.cfg` change modifies the port, the new port will be\noverwritten with the previous one from the state file.\n\n- A workaround is to change the backend server name, so that it is\nconsidered to be a separate server that does not match what is in the state\nfile.\n\n- This has implications for the rollout procedure we use on HAProxy.\n\n\n## Background\n\n\nAll of this occurred in the context of [the gitlab-pages PROXYv2\n\nproject](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11902).\n\n\nThe rollout to staging involves changing the request flow from TCP\nproxying...\n\n```\n                   443                   443                        1443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                        tcp            tcp,tls,http\n```\n\n\n... to using the [PROXY\nprotocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt):\n\n```\n                   443                   443                        2443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                     proxyv2,tcp       proxyv2,tcp,tls,http\n```\n\n\nThis is done through this change to `/etc/haproxy/haproxy.cfg` on\n\n`fe-pages-01-lb-gstg` (note the port change):\n\n```diff\n\n-    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nSeems straightforward enough, let's go ahead and apply that change.\n\n\n## The brokenness\n\n\nAfter applying this change on one of the two `fe-pages` nodes, the requests\nto\n\nthat node start failing.\n\n\nBy retrying a few times via `curl` on the command line, we see this error:\n\n```\n\n➜  ~ curl -vvv https://jarv.staging.gitlab.io/pages-test/\n\n*   Trying 35.229.69.78...\n\n* TCP_NODELAY set\n\n* Connected to jarv.staging.gitlab.io (35.229.69.78) port 443 (#0)\n\n* ALPN, offering h2\n\n* ALPN, offering http/1.1\n\n* successfully set certificate verify locations:\n\n*   CAfile: /etc/ssl/cert.pem\n  CApath: none\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n\n* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n* Closing connection 0\n\ncurl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n```\n\n\nThis looks like some issue in the TLS stack, or possibly with the underlying\n\nconnection. It turns out that `LibreSSL` does not give us much insight into\nthe\n\nunderlying issue here.\n\n\nSo to get a better idea, let's capture a traffic dump on the HAProxy node:\n\n```\n\nsudo tcpdump -v -w \"$(pwd)/$(hostname).$(date +%Y%m%d_%H%M%S).pcap\"\n\n```\n\n\nWhile `tcpdump` is running, we can generate some traffic, then ctrl+c and\npull\n\nthe dump down for further analysis. That `pcap` file can be opened in\nWireshark, and this allows the data to be\n\nexplored and filtered interactively. Here, the first really surprising thing\nhappens:\n\n\n**We do not see any traffic on port 2443.**\n\n\nAt the same time, we _do_ see some traffic on port 1443. But we came here to\nlook at what underlies the LibreSSL error, and what we find\n\nis the following (by filtering for `ip.addr == \u003Cmy external ip>`). We have a\nTCP SYN/ACK, establishing the connection. Followed by the client\n\nsending a TLS \"hello\". After which the server closes the connection with a\nFIN.\n\n\nIn other words, the server is closing the connection on the client.\n\n\n## The early hypotheses\n\n\nSo here come the usual suspects:\n\n\n* Did we modify the correct place in the config file?\n\n* Did we catch all places we need to update in the config?\n\n* Did the HAProxy process parse th econfig successfully?\n\n* Did HAProxy actually reload?\n\n* Is there a difference between reload and restart?\n\n* Did we modify the correct config file?\n\n* Are there old lingering HAProxy processes on the box?\n\n* Are we actually sending traffic to this node?\n\n* Are backend health checks failing?\n\n* Is there anything in the HAProxy logs?\n\n\nNone of these gave any insights whatsoever.\n\n\nIn an effort to reproduce the issue, I ran HAProxy on my local machine with\na\n\nsimilar config, proxying traffic to `web-pages-01-sv-gstg`. To my surprise,\nthis\n\nworked correctly. I tested with different HAProxy versions. It worked\nlocally, but not on\n\n`fe-pages-01`.\n\n\nAt this point I'm stumped. The local config is not identical to gstg, but\nquite\n\nsimilar. What could possibly be the difference?\n\n\n## Digging deeper\n\n\nThis is when I reached out to [Matt Smiley](/company/team#/msmiley) to help\nwith the investigation.\n\n\nWe started off by repeating the experiment. We saw the same results:\n\n\n* Server closes connection after client sends TLS hello\n\n* No traffic from fe-pages to web-pages on port 2443\n\n* Traffic from fe-pages to web-pages on port 1443\n\n\nThe first lead was to look at the packets going to port 1443. What do they\n\ncontain? We see this:\n\n\n![Traffic capture in wireshark showing a TCP FIN and the string QUIT in the\nstream](https://about.gitlab.com/images/blogimages/infra-proxy-protocol-wireshark.png){:\n.shadow.center}\n\nTraffic capture in Wireshark showing a TCP FIN and the string QUIT in the\nstream\n\n{: .note.text-center}\n\n\nThere is mention of `jarv.staging.gitlab.io` which does match what the\nclient sent. And before that there is some really weird preamble:\n\n\n```\n\n\"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\"\n\n```\n\n\nWhat on earth is this? Is it from the PROXY protocol? Let's search [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for the\nword\n\n\"QUIT.\" Nothing.\n\n\nIs this something in the HAProxy source? Searching for \"QUIT\" in the code\n\nreveals some hits, but none that explain this.\n\n\nSo this is a mystery. We leave it for now, and probe in a different\ndirection.\n\n\n## Honing in\n\n\nHow come we are sending traffic to port 1443, when that port is not\nmentioned in\n\n`haproxy.cfg`? Where on earth is HAProxy getting that information from?\n\n\nI suggested running `strace` on HAProxy startup, so that we can see which\nfiles\n\nare being `open`ed. This is a bit tricky to do though, because the process\nis\n\nsystemd-managed.\n\n\nIt turns out that thanks to BPF and [BCC](https://github.com/iovisor/bcc),\nwe\n\ncan actually listen on open events system-wide using the wonderful\n\n[opensnoop](https://github.com/iovisor/bcc/blob/master/tools/opensnoop.py).\nSo we run `opensnoop` and restart `haproxy`, and this is what we see,\nhighlighting the relevant bit:\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo\n/usr/share/bcc/tools/opensnoop  -T --name haproxy\n\n\n...\n\n\n24.117171000  16702  haproxy             3   0 /etc/haproxy/haproxy.cfg\n\n...\n\n24.118099000  16702  haproxy             4   0 /etc/haproxy/errors/400.http\n\n...\n\n24.118333000  16702  haproxy             4   0\n/etc/haproxy/cloudflare_ips_v4.lst\n\n...\n\n24.119109000  16702  haproxy             3   0 /etc/haproxy/state/global\n\n```\n\n\nWhat do we have here? `/etc/haproxy/state/global`, this seems oddly\nsuspicious.\n\nWhat could it possibly be? Let's see what this file contains.\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo cat\n/etc/haproxy/state/global\n\n\n1\n\n# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state\nsrv_uweight srv_iweight srv_time_since_last_change srv_check_status\nsrv_check_result srv_check_health srv_check_state srv_agent_state\nbk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord\n\n5 pages_http 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n5 pages_http 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n6 pages_https 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0\n0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n6 pages_https 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0\n0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n```\n\n\nIt appears we are storing some metadata for each backend server, including\nits old port number!\n\n\nNow, looking again in `haproxy.cfg`, we see:\n\n```\n\nglobal\n    ...\n    server-state-file /etc/haproxy/state/global\n```\n\n\nSo we are using the\n\n[`server-state-file`](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server-state-file)\n\ndirective. This will persist server state across HAProxy restarts. That is\n\nuseful to keep metadata consistent, such as whether a server was marked as\n\nMAINT.\n\n\n**However, it appears to be clobbering the port from `haproxy.cfg`!**\n\n\nThe suspected behavior is:\n\n\n* HAProxy is running with the old config: `web-pages-01-sv-gstg`, `1443`\n\n* `haproxy.cfg` is updated with the new config: `web-pages-01-sv-gstg`,\n`2443`, `send-proxy-v2`\n\n* HAProxy reload is initiated\n\n* HAProxy writes out the state to `/etc/haproxy/state/global` (including the\nold port of each backend server)\n\n* HAProxy starts up, reads `haproxy.cfg`, initializes itself with the new\nconfig: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n\n* HAProxy reads the state from `/etc/haproxy/state/global`, matches on the\nbackend server `web-pages-01-sv-gstg`, and overrides all values, including\nthe port!\n\n\nThe result is that we are now attempting to send PROXYv2 traffic to the TLS\nport.\n\n\n## The workaround\n\n\nTo validate the theory and develop a potential workaround, we modify\n\n`haproxy.cfg` to use a different backend server name.\n\n\nThe new diff is:\n\n```diff\n\n-    server web-pages-01-sv-gstg        \nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg        \nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg-proxyv2\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg-proxyv2\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nWith this config change in place, we reload HAProxy and indeed, it is now\n\nserving traffic correctly. See [the merge request fixing\nit](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/261).\n\n\n## A follow-up on those `QUIT` bytes\n\n\nNow, what is up with that `QUIT` message? Is it part of the PROXY protocol?\nRemember, searching [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for that\n\nstring did not find any matches. However, Matt actually read the spec, and\nfound this section on version 2 of\n\nthe protocol:\n\n```\n\nThe binary header format starts with a constant 12 bytes block containing\nthe\n\nprotocol signature :\n\n   \\x0D \\x0A \\x0D \\x0A \\x00 \\x0D \\x0A \\x51 \\x55 \\x49 \\x54 \\x0A\n```\n\n\nThose are indeed the bytes that make up \"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\". Slightly\nless mnemonic than the header from text-based version 1 of the protocol:\n\n```\n\n- a string identifying the protocol : \"PROXY\" ( \\x50 \\x52 \\x4F \\x58 \\x59 )\n  Seeing this string indicates that this is version 1 of the protocol.\n```\n\n\nWell, I suppose that explains it.\n\n\nI believe our work here is done. Don't forget to like and subscribe!\n","engineering",[24,25],"production","inside GitLab",{"slug":27,"featured":6,"template":28},"this-sre-attempted-to-roll-out-an-haproxy-change","BlogPost","content:en-us:blog:this-sre-attempted-to-roll-out-an-haproxy-change.yml","yaml","This Sre Attempted To Roll Out An Haproxy Change","content","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change.yml","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":449,"_type":30,"title":450,"_source":32,"_file":451,"_stem":452,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":390,"minimal":421,"duo":440},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,201,206,311,371],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":183},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,162],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[149,152,157],{"text":150,"config":151},"Security & Compliance",{"href":145,"dataGaLocation":44,"dataGaName":150},{"text":153,"config":154},"Software Supply Chain Security",{"href":155,"dataGaLocation":44,"dataGaName":156},"/solutions/supply-chain/","Software supply chain security",{"text":158,"config":159},"Compliance & Governance",{"href":160,"dataGaLocation":44,"dataGaName":161},"/solutions/continuous-software-compliance/","Compliance and governance",{"title":163,"link":164,"items":169},"Measurement",{"config":165},{"icon":166,"href":167,"dataGaName":168,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[170,174,178],{"text":171,"config":172},"Visibility & Measurement",{"href":167,"dataGaLocation":44,"dataGaName":173},"Visibility and Measurement",{"text":175,"config":176},"Value Stream Management",{"href":177,"dataGaLocation":44,"dataGaName":175},"/solutions/value-stream-management/",{"text":179,"config":180},"Analytics & Insights",{"href":181,"dataGaLocation":44,"dataGaName":182},"/solutions/analytics-and-insights/","Analytics and insights",{"title":184,"items":185},"GitLab for",[186,191,196],{"text":187,"config":188},"Enterprise",{"href":189,"dataGaLocation":44,"dataGaName":190},"/enterprise/","enterprise",{"text":192,"config":193},"Small Business",{"href":194,"dataGaLocation":44,"dataGaName":195},"/small-business/","small business",{"text":197,"config":198},"Public Sector",{"href":199,"dataGaLocation":44,"dataGaName":200},"/solutions/public-sector/","public sector",{"text":202,"config":203},"Pricing",{"href":204,"dataGaName":205,"dataGaLocation":44,"dataNavLevelOne":205},"/pricing/","pricing",{"text":207,"config":208,"link":210,"lists":214,"feature":298},"Resources",{"dataNavLevelOne":209},"resources",{"text":211,"config":212},"View all resources",{"href":213,"dataGaName":209,"dataGaLocation":44},"/resources/",[215,248,270],{"title":216,"items":217},"Getting started",[218,223,228,233,238,243],{"text":219,"config":220},"Install",{"href":221,"dataGaName":222,"dataGaLocation":44},"/install/","install",{"text":224,"config":225},"Quick start guides",{"href":226,"dataGaName":227,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":229,"config":230},"Learn",{"href":231,"dataGaLocation":44,"dataGaName":232},"https://university.gitlab.com/","learn",{"text":234,"config":235},"Product documentation",{"href":236,"dataGaName":237,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":239,"config":240},"Best practice videos",{"href":241,"dataGaName":242,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":244,"config":245},"Integrations",{"href":246,"dataGaName":247,"dataGaLocation":44},"/integrations/","integrations",{"title":249,"items":250},"Discover",[251,256,260,265],{"text":252,"config":253},"Customer success stories",{"href":254,"dataGaName":255,"dataGaLocation":44},"/customers/","customer success stories",{"text":257,"config":258},"Blog",{"href":259,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":261,"config":262},"Remote",{"href":263,"dataGaName":264,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":266,"config":267},"TeamOps",{"href":268,"dataGaName":269,"dataGaLocation":44},"/teamops/","teamops",{"title":271,"items":272},"Connect",[273,278,283,288,293],{"text":274,"config":275},"GitLab Services",{"href":276,"dataGaName":277,"dataGaLocation":44},"/services/","services",{"text":279,"config":280},"Community",{"href":281,"dataGaName":282,"dataGaLocation":44},"/community/","community",{"text":284,"config":285},"Forum",{"href":286,"dataGaName":287,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":289,"config":290},"Events",{"href":291,"dataGaName":292,"dataGaLocation":44},"/events/","events",{"text":294,"config":295},"Partners",{"href":296,"dataGaName":297,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":299,"textColor":300,"text":301,"image":302,"link":306},"#2f2a6b","#fff","Insights for the future of software development",{"altText":303,"config":304},"the source promo card",{"src":305},"/images/navigation/the-source-promo-card.svg",{"text":307,"config":308},"Read the latest",{"href":309,"dataGaName":310,"dataGaLocation":44},"/the-source/","the source",{"text":312,"config":313,"lists":315},"Company",{"dataNavLevelOne":314},"company",[316],{"items":317},[318,323,329,331,336,341,346,351,356,361,366],{"text":319,"config":320},"About",{"href":321,"dataGaName":322,"dataGaLocation":44},"/company/","about",{"text":324,"config":325,"footerGa":328},"Jobs",{"href":326,"dataGaName":327,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":327},{"text":289,"config":330},{"href":291,"dataGaName":292,"dataGaLocation":44},{"text":332,"config":333},"Leadership",{"href":334,"dataGaName":335,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":337,"config":338},"Team",{"href":339,"dataGaName":340,"dataGaLocation":44},"/company/team/","team",{"text":342,"config":343},"Handbook",{"href":344,"dataGaName":345,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":347,"config":348},"Investor relations",{"href":349,"dataGaName":350,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":352,"config":353},"Trust Center",{"href":354,"dataGaName":355,"dataGaLocation":44},"/security/","trust center",{"text":357,"config":358},"AI Transparency Center",{"href":359,"dataGaName":360,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":362,"config":363},"Newsletter",{"href":364,"dataGaName":365,"dataGaLocation":44},"/company/contact/","newsletter",{"text":367,"config":368},"Press",{"href":369,"dataGaName":370,"dataGaLocation":44},"/press/","press",{"text":372,"config":373,"lists":374},"Contact us",{"dataNavLevelOne":314},[375],{"items":376},[377,380,385],{"text":51,"config":378},{"href":53,"dataGaName":379,"dataGaLocation":44},"talk to sales",{"text":381,"config":382},"Get help",{"href":383,"dataGaName":384,"dataGaLocation":44},"/support/","get help",{"text":386,"config":387},"Customer portal",{"href":388,"dataGaName":389,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":391,"login":392,"suggestions":399},"Close",{"text":393,"link":394},"To search repositories and projects, login to",{"text":395,"config":396},"gitlab.com",{"href":58,"dataGaName":397,"dataGaLocation":398},"search login","search",{"text":400,"default":401},"Suggestions",[402,404,408,410,414,418],{"text":73,"config":403},{"href":78,"dataGaName":73,"dataGaLocation":398},{"text":405,"config":406},"Code Suggestions (AI)",{"href":407,"dataGaName":405,"dataGaLocation":398},"/solutions/code-suggestions/",{"text":125,"config":409},{"href":127,"dataGaName":125,"dataGaLocation":398},{"text":411,"config":412},"GitLab on AWS",{"href":413,"dataGaName":411,"dataGaLocation":398},"/partners/technology-partners/aws/",{"text":415,"config":416},"GitLab on Google Cloud",{"href":417,"dataGaName":415,"dataGaLocation":398},"/partners/technology-partners/google-cloud-platform/",{"text":419,"config":420},"Why GitLab?",{"href":86,"dataGaName":419,"dataGaLocation":398},{"freeTrial":422,"mobileIcon":427,"desktopIcon":432,"secondaryButton":435},{"text":423,"config":424},"Start free trial",{"href":425,"dataGaName":49,"dataGaLocation":426},"https://gitlab.com/-/trials/new/","nav",{"altText":428,"config":429},"Gitlab Icon",{"src":430,"dataGaName":431,"dataGaLocation":426},"/images/brand/gitlab-logo-tanuki.svg","gitlab icon",{"altText":428,"config":433},{"src":434,"dataGaName":431,"dataGaLocation":426},"/images/brand/gitlab-logo-type.svg",{"text":436,"config":437},"Get Started",{"href":438,"dataGaName":439,"dataGaLocation":426},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":441,"mobileIcon":445,"desktopIcon":447},{"text":442,"config":443},"Learn more about GitLab Duo",{"href":78,"dataGaName":444,"dataGaLocation":426},"gitlab duo",{"altText":428,"config":446},{"src":430,"dataGaName":431,"dataGaLocation":426},{"altText":428,"config":448},{"src":434,"dataGaName":431,"dataGaLocation":426},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":454,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":455,"button":456,"image":460,"config":464,"_id":466,"_type":30,"_source":32,"_file":467,"_stem":468,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":84,"config":457},{"href":458,"dataGaName":459,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"altText":461,"config":462},"GitLab Duo Agent Platform",{"src":463},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":465},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":470,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":471,"_id":676,"_type":30,"title":677,"_source":32,"_file":678,"_stem":679,"_extension":35},"/shared/en-us/main-footer",{"text":472,"source":473,"edit":479,"contribute":484,"config":489,"items":494,"minimal":668},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":474,"config":475},"View page source",{"href":476,"dataGaName":477,"dataGaLocation":478},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":480,"config":481},"Edit this page",{"href":482,"dataGaName":483,"dataGaLocation":478},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":485,"config":486},"Please contribute",{"href":487,"dataGaName":488,"dataGaLocation":478},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":490,"facebook":491,"youtube":492,"linkedin":493},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[495,518,575,604,638],{"title":62,"links":496,"subMenu":501},[497],{"text":498,"config":499},"DevSecOps platform",{"href":71,"dataGaName":500,"dataGaLocation":478},"devsecops platform",[502],{"title":202,"links":503},[504,508,513],{"text":505,"config":506},"View plans",{"href":204,"dataGaName":507,"dataGaLocation":478},"view plans",{"text":509,"config":510},"Why Premium?",{"href":511,"dataGaName":512,"dataGaLocation":478},"/pricing/premium/","why premium",{"text":514,"config":515},"Why Ultimate?",{"href":516,"dataGaName":517,"dataGaLocation":478},"/pricing/ultimate/","why ultimate",{"title":519,"links":520},"Solutions",[521,526,529,531,536,541,545,548,552,557,559,562,565,570],{"text":522,"config":523},"Digital transformation",{"href":524,"dataGaName":525,"dataGaLocation":478},"/topics/digital-transformation/","digital transformation",{"text":150,"config":527},{"href":145,"dataGaName":528,"dataGaLocation":478},"security & compliance",{"text":139,"config":530},{"href":121,"dataGaName":122,"dataGaLocation":478},{"text":532,"config":533},"Agile development",{"href":534,"dataGaName":535,"dataGaLocation":478},"/solutions/agile-delivery/","agile delivery",{"text":537,"config":538},"Cloud transformation",{"href":539,"dataGaName":540,"dataGaLocation":478},"/topics/cloud-native/","cloud transformation",{"text":542,"config":543},"SCM",{"href":135,"dataGaName":544,"dataGaLocation":478},"source code management",{"text":125,"config":546},{"href":127,"dataGaName":547,"dataGaLocation":478},"continuous integration & delivery",{"text":549,"config":550},"Value stream management",{"href":177,"dataGaName":551,"dataGaLocation":478},"value stream management",{"text":553,"config":554},"GitOps",{"href":555,"dataGaName":556,"dataGaLocation":478},"/solutions/gitops/","gitops",{"text":187,"config":558},{"href":189,"dataGaName":190,"dataGaLocation":478},{"text":560,"config":561},"Small business",{"href":194,"dataGaName":195,"dataGaLocation":478},{"text":563,"config":564},"Public sector",{"href":199,"dataGaName":200,"dataGaLocation":478},{"text":566,"config":567},"Education",{"href":568,"dataGaName":569,"dataGaLocation":478},"/solutions/education/","education",{"text":571,"config":572},"Financial services",{"href":573,"dataGaName":574,"dataGaLocation":478},"/solutions/finance/","financial services",{"title":207,"links":576},[577,579,581,583,586,588,590,592,594,596,598,600,602],{"text":219,"config":578},{"href":221,"dataGaName":222,"dataGaLocation":478},{"text":224,"config":580},{"href":226,"dataGaName":227,"dataGaLocation":478},{"text":229,"config":582},{"href":231,"dataGaName":232,"dataGaLocation":478},{"text":234,"config":584},{"href":236,"dataGaName":585,"dataGaLocation":478},"docs",{"text":257,"config":587},{"href":259,"dataGaName":5,"dataGaLocation":478},{"text":252,"config":589},{"href":254,"dataGaName":255,"dataGaLocation":478},{"text":261,"config":591},{"href":263,"dataGaName":264,"dataGaLocation":478},{"text":274,"config":593},{"href":276,"dataGaName":277,"dataGaLocation":478},{"text":266,"config":595},{"href":268,"dataGaName":269,"dataGaLocation":478},{"text":279,"config":597},{"href":281,"dataGaName":282,"dataGaLocation":478},{"text":284,"config":599},{"href":286,"dataGaName":287,"dataGaLocation":478},{"text":289,"config":601},{"href":291,"dataGaName":292,"dataGaLocation":478},{"text":294,"config":603},{"href":296,"dataGaName":297,"dataGaLocation":478},{"title":312,"links":605},[606,608,610,612,614,616,618,622,627,629,631,633],{"text":319,"config":607},{"href":321,"dataGaName":314,"dataGaLocation":478},{"text":324,"config":609},{"href":326,"dataGaName":327,"dataGaLocation":478},{"text":332,"config":611},{"href":334,"dataGaName":335,"dataGaLocation":478},{"text":337,"config":613},{"href":339,"dataGaName":340,"dataGaLocation":478},{"text":342,"config":615},{"href":344,"dataGaName":345,"dataGaLocation":478},{"text":347,"config":617},{"href":349,"dataGaName":350,"dataGaLocation":478},{"text":619,"config":620},"Sustainability",{"href":621,"dataGaName":619,"dataGaLocation":478},"/sustainability/",{"text":623,"config":624},"Diversity, inclusion and belonging (DIB)",{"href":625,"dataGaName":626,"dataGaLocation":478},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":352,"config":628},{"href":354,"dataGaName":355,"dataGaLocation":478},{"text":362,"config":630},{"href":364,"dataGaName":365,"dataGaLocation":478},{"text":367,"config":632},{"href":369,"dataGaName":370,"dataGaLocation":478},{"text":634,"config":635},"Modern Slavery Transparency Statement",{"href":636,"dataGaName":637,"dataGaLocation":478},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":639,"links":640},"Contact Us",[641,644,646,648,653,658,663],{"text":642,"config":643},"Contact an expert",{"href":53,"dataGaName":54,"dataGaLocation":478},{"text":381,"config":645},{"href":383,"dataGaName":384,"dataGaLocation":478},{"text":386,"config":647},{"href":388,"dataGaName":389,"dataGaLocation":478},{"text":649,"config":650},"Status",{"href":651,"dataGaName":652,"dataGaLocation":478},"https://status.gitlab.com/","status",{"text":654,"config":655},"Terms of use",{"href":656,"dataGaName":657,"dataGaLocation":478},"/terms/","terms of use",{"text":659,"config":660},"Privacy statement",{"href":661,"dataGaName":662,"dataGaLocation":478},"/privacy/","privacy statement",{"text":664,"config":665},"Cookie preferences",{"dataGaName":666,"dataGaLocation":478,"id":667,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"items":669},[670,672,674],{"text":654,"config":671},{"href":656,"dataGaName":657,"dataGaLocation":478},{"text":659,"config":673},{"href":661,"dataGaName":662,"dataGaLocation":478},{"text":664,"config":675},{"dataGaName":666,"dataGaLocation":478,"id":667,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[681],{"_path":682,"_dir":683,"_draft":6,"_partial":6,"_locale":7,"content":684,"config":688,"_id":690,"_type":30,"title":19,"_source":32,"_file":691,"_stem":692,"_extension":35},"/en-us/blog/authors/igor-wiedler","authors",{"name":19,"config":685},{"headshot":686,"ctfId":687},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681841/Blog/Author%20Headshots/igorwwwwwwwwwwwwwwwwwwww-headshot.png","igorwwwwwwwwwwwwwwwwwwww",{"template":689},"BlogAuthor","content:en-us:blog:authors:igor-wiedler.yml","en-us/blog/authors/igor-wiedler.yml","en-us/blog/authors/igor-wiedler",{"_path":694,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":695,"eyebrow":696,"blurb":697,"button":698,"secondaryButton":702,"_id":704,"_type":30,"title":705,"_source":32,"_file":706,"_stem":707,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":699},{"href":700,"dataGaName":49,"dataGaLocation":701},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":703},{"href":53,"dataGaName":54,"dataGaLocation":701},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1755531303284]