There are a lot of excellent tools in the market for keyword rank tracking– however at Seer we found that we had numerous utilize cases with natural rank information that wasnt being dealt with by those tools.
We dont require day-to-day rankings for this– we need a massive, one-time information pull.
It can also be used in customized analyses by our analytics teams where we join in a clients paid search or CRM data.
They likewise had that information kept in your area in a CSV on their computer. If another team member wanted to develop a different analysis with the very same information, they d have to go through those actions all over once again or ask the very first group member to email them their CSV. They dont need to download the data, tidy and change it, and then build visualizations on top of it.
A lot of our analyses pull big amounts of information (believe 100K, 200K, 500K+ keywords at a time!) for a photo of insights– perhaps were examining content gaps or attempting to look into a brand-new industry for a client. We do not need everyday rankings for this– we require an enormous, one-time data pull.
For some customers, we might have search terms that we want to keep a pulse on, however do we require everyday information? We may wish to blend our tracking– priority keywords daily, other keywords to help us comprehend shifts in the landscape weekly or month-to-month.
If a customer was intending on a migration or a significant modification in their site, we may even want to switch weekly or monthly keywords to track daily so that we can keep a more detailed eye on modifications throughout those high-risk time periods.
& #x 1f4a1; In this post, well walk through how we approached constructing our own keyword tracking tool from the ground-up.
What Are the Benefits of a Custom Keyword Tracking Tool?
The keyword tracking tools we were using didnt give us the flexibility to change tracking frequency– team members had to send large keyword sets and after that carefully keep track of so we might turn off tracking as quickly as information was returned to prevent tracking 500K keywords daily for an analysis that we just required to run two times a year. If an employee forgot to shut off tracking, we could have some pretty high costs for a really simple error.
Lets state an employee wished to send 50,000 search terms for a snapshot analysis, which the expense of our old tool was ~$ 400/day for 50K keywords.
Using our old tool, our employee would submit their keywords, then every day they d examine to see when the data had actually returned– generally within ~ 3-5 days. Then they d need to shut off tracking instantly and download the data, or those keywords would continue to incur expenses.
What if one of that staff members clients had a significant issue and they needed to jump in to assist them out? It would be easy (and understandable) to forget to turn off tracking if that team member had a major analysis and discussion turning up, and then a client fire that they required to deal with on top of it. (Christina can vouch for this– shes been among those group members who forgot to turn off tracking prior to).
Using our old tools, those keywords would continue sustaining expenses to the tune of $400/day till that group member kept in mind to turn them off or if among our team members who was monitoring for significant overages captured it. Now use that threat to an entire team. Human mistakes happen– and something we might do to help our group is removing the responsibility of keeping in mind to turn off keywords.
When analyzing snapshot (one-time rankings) keywords from our old tool, we discovered that we were tracking (and spending for) 4-5x the rankings we actually required, primarily due to the fact that information often took several days to return and sometimes employee didnt switch off tracking instantly. Around 80% of those rankings were redundant.
Count of snapshot keywords
Count of deduplicated picture keywords
% of waste
With our own keyword tracking tool, that exact same team member submits their 50K search terms utilizing the “one-time” frequency and thats it. When their data is available, they can utilize it– no extra steps required.
Were also able to submit more keywords by partnering with Traject Data than ever in the past– we went from 26M rankings in 2019 to 61M rankings in 2020, the year we launched our internal rank tracking tool.
The day this article was written (2/10/22) we had 4.6 billion rows of keyword data in our rank tracking information lake.
Of all of the processed rankings with Traject, we see the following frequencies:.
Count of tracked keywords.
% to total.
Thats over 80% of all overall search terms tracked at a one-time frequency in the previous 2 years! Envision all of the time saved and cost overages avoided by giving users the flexibility to track at different frequencies.
When we were still utilizing other rank tracking tools, we d need to export the data if we wanted to personalize anything outside of their software application. Even if we built templates in visualization tools like Power BI or Google Data Studio to speed up build time, those design templates would anticipate very specific inputs– if a tool altered the name of a column or if a staff member following a list of steps to export data missed out on a step, it might cause errors and confusion.
A team member performing an analysis might follow 10-20 instructions for exporting the information a particular way– possibly it calls for a specific report or filtering the information a particular way before exporting. The employee exports their CSV to their computer system, opens up the design template, and selects their CSV as a source.
All of the sudden, everything breaks and they are hit with a glaring error message– “The column “Landing Page” wasnt discovered.” All of the other data sources in the analysis stop working too– due to the failure from the missing out on “Landing Page” column.
They jump into a chat and ask some other team members for assistance. Another team member states they can leap on a video chat to assist troubleshoot.
Finally, someone who ran into this issue prior to asks our initial employee to open the CSV they downloaded. The “Landing Page” column isnt in the CSV– it has actually been changed by the field name “URL”.
” Sometimes this tool just alters column names and we dont learn up until it breaks something– Ill tap the individual who owns the design template and directions to make an update” our more seasoned employee states.
Issue solved, however this procedure took numerous staff member maybe 30 minutes to fix it. Our original staff member didnt desire to squander anyones time– they might have spent an hour trying to repair it themselves prior to they even asked for help.
By managing our information we guarantee that modifications like that dont take place to team members. Even if a vendor makes a modification, we can “conceal” those types of modifications from the group in our change layer– maybe relabeling the “URL” column back to “Landing Page” in a cleaning action before that information gets into the hands of our group.
Keeping our data in our storage facility also provides us the opportunity to reuse information. We might have multiple products that have a data source in typical– if those could utilize the very same information, we might create more value without increasing costs.
We can also multiply those cost savings through creating queues to microservices that deduplicate and cache data– increasing and decreasing expenses turn-around times for group members to get data.
Our data can flow into scalable dashboards and reports for insights that every SEO staff member wishes to know (like “how has my rank for priority keywords changed week over week?” or “how did our rank improve after implementing that content audit?”) however it can also be used in custom-made analyses by our analytics teams where we take part a clients paid search or CRM data.
After just a couple of months of launching our internal rank tracking tool we began to strike information size restrictions in our visualization tool, and with our data growing greatly we needed to transfer to a data platform that could query petabytes vs gigabytes. We had the ability to provide ourselves some runway by carrying out incremental refreshes or by straining information that wasnt a must-have in each control panel (which triggered extra time for developing each data product).
At the end of 2021, we migrated our data products (including rank tracking data) from one data platform to our own web application powered by Lookers embedded analytics..
Because the information remained in our storage facility, we were able to change the information and rebuild utilizing best practices for our brand-new information platform. We kept our old platform running till our new platform was ready to release– something that may not have been possible without the capability to use the same data in several applications.
A significant worth of creating our own rank tracking tool is the capability to build robust security into the system, not only by keeping our information safe in our warehouse but by utilizing consents to produce a better experience for our staff member.
By joining our customers natural ranking data with information from our CRM (like “which employee is designated to what client”) we can utilize permissioning to make sure that customer marketing data is only visible to group members dealing with that client. For a Seer staff member, when they open up one of our information items, they only see their customers data, making navigating through products much easier.
This also provides us opportunities to anonymize data and integrate it for industry-level trends and insights, without counting on manual approaches more prone to user-error.
Having our own rank tracking tool suggests that all of that information streams into our data storage facility, where we can direct it several ways for our group to consume it– we can equalize our information by allowing our staff member to make data-informed choices and feel great about data, no matter their technical expertise.
Remember our employee with the CSV concern?
They likewise had that data saved locally in a CSV on their computer. If another employee desired to develop a different analysis with the exact same data, they d have to go through those steps all over once again or ask the first string member to email them their CSV. All of that work and data is decentralized.
Storing data in our warehouse provides us the flexibility to grant staff member access to the exact same central data in various formats. Information items that scale to the whole team might utilize information lakes, however we also develop bite-sized tables and curated views that might be envisioned in Power BI, Google Data Studio, Tableau, or any other tool a staff member may wish to utilize– we dont require them into specific tool however motivate group members to use whatever tools they are positive utilizing.
This likewise reduces training and build time– for reports that every team member no matter experience or role should have access to, well create a control panel that they just click to open. They do not need to download the information, clean and transform it, and after that develop visualizations on top of it. They simply login and open the dashboard and voila– its there!
For staff member who construct custom dashboards as part of their function, well offer them with structured information that can assist them quickly develop the foundation of their analysis, and then they utilize their skills to tailor it.
How Has Building Our Own Rank Tracking Product Changed How We Work?
In 2019, our group acquired 24M rankings at a day-to-day frequency. We tracked 12M photo rankings, but ~ 80% were redundant (only 2.4 M unique rankings).
In the first half of 2020, we got 14M rankings at a day-to-day frequency and another 5M snapshot rankings (once again, ~ 80% of picture rankings were redundant– we just needed 1M). After we migrated to our rank tracking tool in July 2020, we processed 46M total rankings across several frequencies.
In 2021, we returned 56M rankings. 76% of overall rankings were run on a one-time frequency. Just 9% of those keywords were tracked daily.
Were just a few weeks into 2022 and weve returned 4.6 M rankings so far, with 65% of rankings tracked at a one-time frequency. We can process approximately 1M keywords per day, which information streams into several products and tools across our central platform.
And not a single employee is doing daily checks to see if their information was returned so that they can switch off tracking.
Wish to Work With Bigger Data?
If youre a natural-born expert who likes resolving issues at scale utilizing huge data– check out Seer Careers & & apply now (were working with folks similar to you!).
SIGN UP WITH OUR TEAM.
Register for our newsletter for more posts like this in your inbox:.
This post was written by: Christina Blake & & Ethan Lyon.
Why Develop a Customized Rank Tracking Tool?