Hi, guys. A quick video on server log analysis. Now, I'm sure many of you, first and foremost, would know where to get your server log's first. So I'm gonna show you where to get in if you've got your cPanel account or whatever. What you need to do before you do any server log analysis is go to metrics and go to row access, what you'll find is you've archived your logs and you've got a bunch of different files here based on your own website. Obviously, I've got November's logs here. So I can click here, download these logs from our website, and there's quite a lot of them there. Yeah, that's where you get them from, first and foremost.
Now, obviously, server log analysis was initially a technical thing that a lot of people didn't know how to do. One of the old ways to do it was use Screaming Frog as your log final analyzer. You can download that, which was fairly low cost. I believe there was a free version, AP version. So there are free and paid versions of Screaming Frog. You can upload your log files there, analyze them, see what bots are happening on your pages and whatnot. So that all works well, and that's pretty much what a server log analysis guy would do for you. There are various other tools, and some people get there own bespoke stuff, and various other bits and bobs.
But, one of the main tools out there is SEMrush. They now have a log file analyzer. It's in beta mode just now, so it's still being developed and they are still messing about with it. If you just go down to the second bottom option on here that plays and you are able to then play about with your server logs and upload them to SEMrush, which then gives you them in a nice, clean, easy-to-understand platform. So what you can do is, obviously, download them from the row access there. You obviously want to uncompress them, so you just click on the compressed version and it will open up, and then you drop your files in. So that was my files there, so I click open and drop the files in there.
The tool does take 5 or 10 minutes to analyze your logs, and then it spits out a report that looks a lot like this. This is obviously my server log analysis from November. You can see the bar, this data's called file type. You can obviously see all of the pages here, and you can filter all of this stuff down to successful pages, file types, and various other bits and bobs. Obviously, the path ... You'll obviously see the path here, so you can obviously see the bots I had in my plug-ins, and stuff like that is where you can see all this data so you can break it all down. The file paths are all here. The bot hacks are there, share of the hacks' percentages there. Obviously, what we are ... When I say we, I mean SEMrush, I invited 40 people and we worked with a product developer and this was one of the tools that we worked on. The majority of wanted to see the crawl frequency and the last crawl. That's what was asked for, and the last status as well. So you can see here, crawl frequency every 20 minutes. It was last crawled at this time here. You can obviously see the last status called on them as well, obviously 200 is a good thing. But, yeah, you obviously want to work your way through and analyze your server logs.
You can do all server logs, Google desktop bot, or Google smartphone, so you can check those out. You can't obviously check other bots at present. That is something that may or may not come out in the future, you'd have to ask SEMrush that part. But, I think for the vast majority of people out there, the server logs is the one that's most important. It's the one that we all want to and Google, we all want to make sure that the Googlebot likes our website and can get on about a page, and it can frequently crawl those pages, and stuff like that, and what you see what's getting on there. You can obviously filter all of this information down by file type. If it was HTML files you were wanting to check out, then you can check all them out there. Obviously, the HTML files are all making a blog post and whatnot on your website. We can see that some of those are redirected and some of them are just pages that are working well. But, you can obviously filter all that down for yourself, and you can obviously filter it by path. But, that is entirely up to yourself and how deep you want to analyze your server logs. It's great to find out how the bots are hacking your website.
I think, obviously, for me, who is not by any stretch of the imagination, a server log analysis expert. It's not something that I obsess over, but there are certainly guys out there. One, for example, is Andy Halliday who is a geek on this type of stuff and is obsessed with it, and is an expert on server logs. I think Don Anderson's another one. These guys are right into the server logs, check that stuff out on a regular ongoing basis, because obviously, you can identify problems with Googlebot hacking your pages and stuff like that, and obviously make adjustments to your website as a result of that. But, what I find really helpful is having it in a nice, clean, easy-to-understand dashboard, which is what SEMrush has brought into play. Obviously, very easy for someone who's not as technical, as some of the other one's that I've mentioned, to be able to understand and implement changes to the website as a result of these server logs.
So server logs is not a big thing, amazing that SEMrush has brought this out. If you're not a SEMrush subscriber, you can do it the old fashioned way through Screaming Frog, and if you Google some other stuff, there are a lot of places you can upload your server logs to that will allow you to do server log analysis. But, for me, SEMrush is a tool that many agencies, or freelancers, or whatever you may be, will be using, and it's just another great feature that SEMrush has added on top of all the other stuff you can see here that's all been added.
That is how you do server log analysis. Obviously, I can't show you individual websites and stuff like that, but obviously you want the Googlebot to be able to freely crawl your website and not find broken pages and all that kind of stuff. So that's the end goal, so the line is where you want to be at. Obviously, you can see it's up and down, up and down, up and down for one reason or another. But, that's obviously something that we always look to strive to better our own websites. Hopefully, that gives you a bit more of an update on server log analysis, how you can analyze your files, and also, where you can find your server logs, because a lot of people don't even know where they can find that. This is where you can find your server logs within your control panel. So I hope that helps. Cheers.