Greensboring.com | Greensboro, NC

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-22T22:56:59+00:00

Yes, because instead of using just one request to pull all IDs, it cycles through each forum. The main reason for that limit originally was that some users were getting errors in Google Webmaster Tools past 50000 URLs (apparently their max). This just works the same, but might require some math. Technically I think it's a bit more versatile this way too... and even better is that it works for you! So that's awesome!

Statistics: Posted by Liv — Fri Mar 22, 2013 10:56 pm

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-22T22:53:51+00:00

I think it works. No errors and it shows more topics. So the urls=5000 is now a per forum setting?

Thanks.

Statistics: Posted by htmanning — Fri Mar 22, 2013 10:53 pm

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-22T22:44:37+00:00

Okay... try this....

I went ahead and broke apart the sql request for the topics by forum, so that each forum runs separately. If it is a sql issue it should run much better for you and expand the number of topics you can use.

If it doesn't work, then there's a limit per php file to sql.

Let me know if it works, and if so I'll release it for others to use.

Statistics: Posted by Liv — Fri Mar 22, 2013 10:44 pm

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-20T07:49:00+00:00

Yeah, I'm keeping the 7000 setting. I never thought of having more than one sitemap. How would I write the sql line to query everything after the first 7000?

I've installed this on 4 forums, and 3 worked well. On the fourth I kept getting the copyright credit error. I checked the installation and it was just as the others were. No matter what I tried I kept getting the error. I finally had to remove the IF statement that causes it to search for the credit link just to get it to run. You have any ideas? I'm stumped.

I left the credit to your site in there, but just removed the IF statement that searches for it. Maybe I'm missing something, but I don't think so.

Thanks for your great work on this mod. It works great.

Statistics: Posted by htmanning — Wed Mar 20, 2013 7:49 am

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-20T01:05:09+00:00

Could be a sql timeout of some sort, that limits the number of records that can be pulled at a single time from a Select statement (how it pulls the data)

I'll keep this in mind if I make a future version and have some free time. Might be possible to pause the calls to the server, but I'm unfortunately swamped with mid-terms at the University right now to really get into it.

I'd keep the 7,000 URL setting, and I'll keep you posted in the future if I get a chance to update it.

It could be a memory issue too. I believe SQL will put the data called into memory, before it dumps it into the script, and once that's full then there's not much it can do.

In either case, I think the work around would be to have multiple sitemaps, it's just I've never had to deal with it personally, so I've never coded it (I think my largest database was 7-9000 topics).

Then again if you do have these rogue topics (that don't really exist in the database), a repair of that issue might solve both issues.

Best thing might be to do is find one of these topic numbers, and check in your SQL dashboard if the topics do actually exist with posts too. If not, I'm certain someone on PHPBB would be glad to write you a $15 code to clean that up.

Statistics: Posted by Liv — Wed Mar 20, 2013 1:05 am

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-19T07:51:53+00:00

I checked the max_execution_time on my server and it's currently 30 seconds. The sitemap.php can't be taking more than 3 or 4 seconds so I don't think that's the issue. I can only set it at 7000 urls or below or the script will choke. If you think of anything else please let me know.

Thanks.

Tom

Statistics: Posted by htmanning — Tue Mar 19, 2013 7:51 am

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-18T20:15:14+00:00

Looks like you're on to something. Anything more than $urls=7000 and it chokes.

Statistics: Posted by htmanning — Mon Mar 18, 2013 8:15 pm

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-18T20:02:34+00:00

My guess is it's still an execution time-out, by your host.... try a number like 5,000 and see what happens, then creep up from there in 5 or 10,000 increments till you max out your servers time out parameter.

The good news is that it does list the newest topics first, so it still should work wonders for you if that works.

As far as the phantom topics, there's two tables in PHPBB that topics come from (basically)... the Topics_Table and the Posts_Table.

It is possible that you have information in the Topics_Table (where we draw the data from for this sitemap) but not the Posts_Table.

The only way I can imagine this happening is by a modification or table merger sometime in the past. If this is the case, you can either choose to fix it, or ignore it, as it shouldn't really matter as far as the modification goes. Even if you get errors, it will still function for Google (though fixing the time-out issue is probably important).

I guess the first place to start is check the topics in question, and if they don't actually exist (meaning you can't see them with admin privileges), then something is up with your database, and you'll likely want to run a script to clean out those rogue entries (if you want to put that much effort into it).

If they do exist, then LMK if I missed something, because I'd definitely love to check and see what's going on.

Statistics: Posted by Liv — Mon Mar 18, 2013 8:02 pm

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-18T23:56:28+00:00

Thanks for the quick response.

I adjusted the $urls number earlier and that made no difference. It still stops at the same line. I currently have it set to 100,000 although I also tried 70,000 and it made no difference.

I have one Announcements forum that only the admin can post for, but I have that same forum at other installs and it doesn't cause a problem. However, I did as you suggested and turned off indexing. Sitemap.php now stops at the same line number (21620), but at a different record. It was stopping at topic_id 901, but after turning off the indexing for that one forum it goes until topic_id 884.

1969 error: I seem to have a few topics in the database that were spam, and were removed by a moderator, but for some reason the title of the topic is still there (no post). So I'm wondering if maybe there is no corresponding entry in the database that would hold the body of the post.

Thanks.

Statistics: Posted by htmanning — Mon Mar 18, 2013 7:31 pm

Re: Sitemap for Google - PHPBB 3 Mod

2013-03-18T19:20:40+00:00

If you have to many URLs that your server stops responding on the script, you can definitely adjust the number of URLs to a more reasonable level. This feature was added a few versions ago, so assuming you have the latest, you'd simply adjust this parameter:

Code:: $urls=50000;

(Default is 50,000)

As far as the first question, I'm not sure how you have topics in the database but are saying the topic doesn't exist. There's no real option to do that in PHPBB (to my knowledge [if there is I'd be glad to add that into a future version, but I don't see it in the tables]).

Now if the posts are in a particular forum (f=) which is private, then it's possible you could get the error because the topic wouldn't exist for search engines.

In that case you'd simply make sure that in your control panel, under the private forums you've set ENABLE SEARCH INDEXING to NO for those forums, and they'll automatically not appear in the sitemap.

Go to FORUMS > EDIT FORUM > GENERAL FORUM SETTINGS

Code:: Enable search indexing: If set to yes posts made to this forum will be indexed for searching.

and set it to NO for private forums.

This was also a feature that's built into the last few versions.

Statistics: Posted by Liv — Mon Mar 18, 2013 7:20 pm