Making a English Wikipedia server - Revision history

Sam: /* Finish configuring LocalSettings.php */ Removed unneeded lines

2024-10-10T03:47:08Z

Finish configuring LocalSettings.php: Removed unneeded lines

← Older revision		Revision as of 03:47, 10 October 2024
Line 129:		Line 129:
	* Add a line to enable a favicon:		* Add a line to enable a favicon:
	$wgFavicon = "$wgResourceBasePath/resources/assets/<logoimage.svg>";		$wgFavicon = "$wgResourceBasePath/resources/assets/<logoimage.svg>";
	* Change <code>$wgEnableUserEmail</code> to <code>false</code>
	* Edit <code>$wgEmergencyContact</code> and <code>$wgPasswordSender</code> with your email address between the quote marks		* Edit <code>$wgEmergencyContact</code> and <code>$wgPasswordSender</code> with your email address between the quote marks
	* Change <code>$wgLocaltimezone</code> from UTC to your correct PHP time zone - in our case, <code>Europe/London</code>		* Change <code>$wgLocaltimezone</code> from UTC to your correct PHP time zone - in our case, <code>Europe/London</code>
Line 165:		Line 164:
	'url' => 'https://commons.wikimedia.org/w/api.php'		'url' => 'https://commons.wikimedia.org/w/api.php'
	];		];
	* Add under <code>wfLoadExtension( 'Math' );</code>:
	~~$wgDefaultUserOptions['math'] = 'mathml';~~
	* Add under <code>wfLoadExtension( 'MobileFrontend' );</code>:		* Add under <code>wfLoadExtension( 'MobileFrontend' );</code>:
	~~$wgMFAutodetectMobileView = true;~~
	$wgMFDefaultSkinClass = 'SkinMinerva';		$wgMFDefaultSkinClass = 'SkinMinerva';
	* Add under <code>wfLoadExtension( 'ParserFunctions' );</code>:		* Add under <code>wfLoadExtension( 'ParserFunctions' );</code>:
Line 174:		Line 170:
	* Add under <code>wfLoadExtension( 'Scribunto' );</code>:		* Add under <code>wfLoadExtension( 'Scribunto' );</code>:
	$wgScribuntoDefaultEngine = 'luastandalone';		$wgScribuntoDefaultEngine = 'luastandalone';
	~~$wgScribuntoEngineConf['luastandalone']['cpuLimit'] = '60'; # Allows lua scripts to run for 60 seconds - needed during import~~
	* Add the following block at the end (leave commented out until needed for debugging for security and comment back out once debugging finished):		* Add the following block at the end (leave commented out until needed for debugging for security and comment back out once debugging finished):
	# Show PHP exceptions (only enable on error)		# Show PHP exceptions (only enable on error)

Sam: /* Finish configuring LocalSettings.php */ Added extra option for ParserFunctions

2024-08-13T23:52:43Z

Finish configuring LocalSettings.php: Added extra option for ParserFunctions

← Older revision		Revision as of 23:52, 13 August 2024
Line 170:		Line 170:
	$wgMFAutodetectMobileView = true;		$wgMFAutodetectMobileView = true;
	$wgMFDefaultSkinClass = 'SkinMinerva';		$wgMFDefaultSkinClass = 'SkinMinerva';
	* Add under <code>wfLoadExtension( 'Scribunto' );</code>		* Add under <code>wfLoadExtension( 'ParserFunctions' );</code>:
			$wgPFEnableStringFunctions = true;
			* Add under <code>wfLoadExtension( 'Scribunto' );</code>:
	$wgScribuntoDefaultEngine = 'luastandalone';		$wgScribuntoDefaultEngine = 'luastandalone';
	$wgScribuntoEngineConf['luastandalone']['cpuLimit'] = '60'; # Allows lua scripts to run for 60 seconds - needed during import		$wgScribuntoEngineConf['luastandalone']['cpuLimit'] = '60'; # Allows lua scripts to run for 60 seconds - needed during import

Sam: /* Install Mediawiki */ Added ParserFunctions to extensions list

2024-08-13T23:49:26Z

Install Mediawiki: Added ParserFunctions to extensions list

← Older revision		Revision as of 23:49, 13 August 2024
Line 114:		Line 114:
	*** JsonConfig		*** JsonConfig
	*** MobileFrontend		*** MobileFrontend
			*** ParserFunctions
	** Check 'Enable Instant Commons'		** Check 'Enable Instant Commons'
	** Change under Logo (icon) and Sidebar logo (optional) <code>change-your-logo.svg</code> to the filename of the image you dropped in the resource folder earlier. You will see a preview of it underneath		** Change under Logo (icon) and Sidebar logo (optional) <code>change-your-logo.svg</code> to the filename of the image you dropped in the resource folder earlier. You will see a preview of it underneath

Sam: /* Finish configuring LocalSettings.php */ Added large article part to LocalSettings.php

2024-08-13T19:53:05Z

Finish configuring LocalSettings.php: Added large article part to LocalSettings.php

← Older revision		Revision as of 19:53, 13 August 2024
Line 175:		Line 175:
	# Show PHP exceptions (only enable on error)		# Show PHP exceptions (only enable on error)
	#$wgShowExceptionDetails = true;		#$wgShowExceptionDetails = true;
			* Add the following block at the end - this allows larger modules to be uploaded without errors:
			$wgMaxArticleSize = 4096; # Size in kb
			$wgAPIMaxResultSize = 4096000; # Size in b

	Go back to the website and check that it is still functional. If you are greeted by a blank page, the most common reason is a missing <code>;</code> at the end of a line - all lines must end with a <code>;</code>.		Go back to the website and check that it is still functional. If you are greeted by a blank page, the most common reason is a missing <code>;</code> at the end of a line - all lines must end with a <code>;</code>.

	== Importing the dump ==		== Importing the dump ==

Sam: Spell checked

2024-08-12T02:53:12Z

Spell checked

← Older revision		Revision as of 02:53, 12 August 2024
Line 1:		Line 1:
	[https://www.mediawiki.org MediaWiki] makes dumps of the English Wikipedia about once a month. As it is free and open source content, you can use these dumps to make your own server with the English Wikipedia content in it. As the English Wikipedia is the largest Wikipedia, it does take a while to import the dumps, but it is by no means impossible. This guide will help you long the way.		[https://www.mediawiki.org MediaWiki] makes dumps of the English Wikipedia about once a month. As it is free and open source content, you can use these dumps to make your own server with the English Wikipedia content in it. As the English Wikipedia is the largest Wikipedia, it does take a while to import the dumps, but it is by no means impossible. This guide will help you long the way.

	This guide is based ~~entierly~~ on the English Wikipedia, but should be quite ~~relevent~~ for other languages too.		This guide is based entirely on the English Wikipedia, but should be quite relevant for other languages too.

	== Prerequisites ==		== Prerequisites ==
Line 27:		Line 27:
	* Prepare MySQL/MariaDB and PHP for the incoming large transactions.		* Prepare MySQL/MariaDB and PHP for the incoming large transactions.

	== ~~Preperation~~ ==		== Preparation ==

	=== Downloading the dumps ===		=== Downloading the dumps ===
Line 33:		Line 33:
	# The dumps for English Wikipedia are available from [https://dumps.wikimedia.org/enwiki/ Wikimedia Dumps]. When there, you'll obviously want to select the latest date.		# The dumps for English Wikipedia are available from [https://dumps.wikimedia.org/enwiki/ Wikimedia Dumps]. When there, you'll obviously want to select the latest date.
	# Once there, you'll need to download the following:		# Once there, you'll need to download the following:
	#* <code>enwiki-(date)-pages-articles-multistream.xml.bz2</code> (This is the latest revision of every Wikipedia page, article and template - the basics you need to get going)		#* <code>enwiki-<date>-pages-articles-multistream.xml.bz2</code> (This is the latest revision of every Wikipedia page, article and template - the basics you need to get going)
	# This is compressed with Bzip2 - if you have the space, extract it once downloaded to speed up importing.		# This is compressed with Bzip2 - if you have the space, extract it once downloaded to speed up importing.

Line 67:		Line 67:
	This can be tweaked with different database and user names as required.		This can be tweaked with different database and user names as required.

	=== Moving ~~databse~~ to a different hard drive ===		=== Moving database to a different hard drive ===

	Due to the sheer size of the ~~databse~~, you may choose to move the MariaDB database to a different drive. MariaDB stores each database in a ~~seperate~~ folder by default making this easy.		Due to the sheer size of the database, you may choose to move the MariaDB database to a different drive. MariaDB stores each database in a separate folder by default making this easy.

	* Stop MariaDB		* Stop MariaDB
Line 119:		Line 119:
	* Click 'Continue'		* Click 'Continue'
	* Click 'Continue' to finish the install		* Click 'Continue' to finish the install
	* Ensure the installation has completed ~~sucessfully~~ then click 'Continue'		* Ensure the installation has completed successfully then click 'Continue'
	* A copy of the LocalSettings.php file will be downloaded automatically. This needs to be moved to the root folder of the wiki which will then make it work.		* A copy of the LocalSettings.php file will be downloaded automatically. This needs to be moved to the root folder of the wiki which will then make it work.
	* Click 'Enter your wiki'. You will be greeted with the default MediaWiki Main page if all is well.		* Click 'Enter your wiki'. You will be greeted with the default MediaWiki Main page if all is well.
Line 130:		Line 130:
	* Change <code>$wgEnableUserEmail</code> to <code>false</code>		* Change <code>$wgEnableUserEmail</code> to <code>false</code>
	* Edit <code>$wgEmergencyContact</code> and <code>$wgPasswordSender</code> with your email address between the quote marks		* Edit <code>$wgEmergencyContact</code> and <code>$wgPasswordSender</code> with your email address between the quote marks
	* Change <code>$wgLocaltimezone</code> from UTC to your correct PHP ~~timezone~~ - in our case, <code>Europe/London</code>		* Change <code>$wgLocaltimezone</code> from UTC to your correct PHP time zone - in our case, <code>Europe/London</code>
	* ~~Underneath~~ <code>$wgGroupPermissions['*']['edit'] = false;</code>~~, add the following~~		* Add under <code>$wgGroupPermissions['*']['edit'] = false;</code>:
	$wgGroupPermissions['*']['createaccount'] = false;		$wgGroupPermissions['*']['createaccount'] = false;
	This stops user accounts being created and prevents editing by anyone other than yourself.		This stops user accounts being created and prevents editing by anyone other than yourself.

Sam: /* Downloading and installing MediaWiki */ Added extra extensions

2024-08-12T02:48:17Z

Downloading and installing MediaWiki: Added extra extensions

← Older revision		Revision as of 02:48, 12 August 2024
Line 45:		Line 45:
	* Copy the folder to the webroot		* Copy the folder to the webroot
	* Place a file in the .../resources/asset folder if using a picture for the site logo/favicon		* Place a file in the .../resources/asset folder if using a picture for the site logo/favicon
			* The following extra extensions need to be downloaded, extracted and dropped in the extensions folder:
			** [https://www.mediawiki.org/wiki/Extension:MassMessage MassMessage]
			** [https://www.mediawiki.org/wiki/Extension:TemplateStyles TemplateStyles]
			** [https://www.mediawiki.org/wiki/Extension:JsonConfig JsonConfig]
			** [https://www.mediawiki.org/wiki/Extension:MobileFrontend MobileFrontend]

	=== Preparing the database ===		=== Preparing the database ===

Sam: /* Finish configuring LocalSettings.php */ Added check website

2024-08-12T02:44:21Z

Finish configuring LocalSettings.php: Added check website

← Older revision		Revision as of 02:44, 12 August 2024
Line 170:		Line 170:
	# Show PHP exceptions (only enable on error)		# Show PHP exceptions (only enable on error)
	#$wgShowExceptionDetails = true;		#$wgShowExceptionDetails = true;

			Go back to the website and check that it is still functional. If you are greeted by a blank page, the most common reason is a missing <code>;</code> at the end of a line - all lines must end with a <code>;</code>.

	== Importing the dump ==		== Importing the dump ==

Sam: /* Preperation */ Added LocalSettings.php additions

2024-08-12T02:42:41Z

Preperation: Added LocalSettings.php additions

@@ Line 117: / Line 117: @@
 * A copy of the LocalSettings.php file will be downloaded automatically. This needs to be moved to the root folder of the wiki which will then make it work.
 * Click 'Enter your wiki'. You will be greeted with the default MediaWiki Main page if all is well.
 == Importing the dump ==

Sam: Saved progress so far

2024-08-12T02:28:23Z

Saved progress so far

@@ Line 15: / Line 15: @@
 * MySQL/MariaDB
 * A dump of your Wikipedia of choice
 === Before you start ===
@@ Line 72: / Line 78: @@
 === Install Mediawiki ===
-* Navigate to where your instance is installed: for example, https://enwiki.freddythechick.net/. You will be greeted by the Mediawiki installer.
+* Navigate to where your instance is installed: for example, https://enwiki.freddythechick.net/. You will be greeted by the Mediawiki installer. Click the 'set up the wiki' link
+* Choose your language then click 'Continue'
 == Importing the dump ==

Sam: Saved progress so far

2024-08-12T01:55:45Z

Saved progress so far

← Older revision		Revision as of 01:55, 12 August 2024
Line 1:		Line 1:
	[https://www.mediawiki.org MediaWiki] makes dumps of the English Wikipedia about once a month. As it is free and open source content, you can use these dumps to make your own server with the English Wikipedia content in it. As the English Wikipedia is the largest Wikipedia, it does take a while to import the dumps, but it is by no means impossible.		[https://www.mediawiki.org MediaWiki] makes dumps of the English Wikipedia about once a month. As it is free and open source content, you can use these dumps to make your own server with the English Wikipedia content in it. As the English Wikipedia is the largest Wikipedia, it does take a while to import the dumps, but it is by no means impossible. This guide will help you long the way.
	This guide will help you long the way.
			This guide is based entierly on the English Wikipedia, but should be quite relevent for other languages too.

	== Prerequisites ==		== Prerequisites ==
Line 8:		Line 9:
	=== What you need to know ===		=== What you need to know ===

	You will need the following to get started ~~with a dump~~:		You will need the following to get started:

	* Apache Web Server		* Apache Web Server
Line 14:		Line 15:
	* MySQL/MariaDB		* MySQL/MariaDB
	* A dump of your Wikipedia of choice		* A dump of your Wikipedia of choice
	* <s>MWDumper</s>
	* <s>The latest Java JRE and Java JDK from the [https://www.java.com Oracle Java] website</s>

	=== Before you start ===		=== Before you start ===

	* Remember that some of the Wikipedia dumps are huge. You will need a lot of disk space (Around ~~200GB~~ as a minimum, around 500GB to do this comfortably, remember that Wikipedia is always growing...)		* Remember that some of the Wikipedia dumps are huge. You will need a lot of disk space (Around 100GB as a minimum for the download, around 500GB to do this comfortably once the database grows, remember that Wikipedia is always growing...)
	* Prepare MySQL/MariaDB for the large transactions coming up. A suggestion is to look in <code>/usr/share/mysql/my-huge.cnf</code> and consider using the values under the <code>[mysql]</code> header, at least while you are importing the database. The most important value that '''MUST''' be changed or the import will fail, is the value <code>max_allowed_packet = 1M</code>. This will need to be changed to <code>max_allowed_packet = 128M</code>. Due to the size of some of Wikipedia's articles, if this value isn't changed, MySQL/MariaDB will reject the record if it is more than 1M and the size isn't changed. 128M is more than enough during import and can be safely changed back after.		* Prepare MySQL/MariaDB and PHP for the incoming large transactions.
	* The tables must be cleared as per the instructions below before attempting importing or it will fail.

	== ~~Importing the dump~~ ==		== Preperation ==

	=== Downloading the dumps ===		=== Downloading the dumps ===

	# The dumps for English Wikipedia are available from [https://dumps.wikimedia.org/enwiki/ ~~here~~]. When there, you'll obviously want to select the latest date.		# The dumps for English Wikipedia are available from [https://dumps.wikimedia.org/enwiki/ Wikimedia Dumps]. When there, you'll obviously want to select the latest date.
	# Once there, you'll need to download the following:		# Once there, you'll need to download the following:
	#* <code>enwiki-<date>-pages-~~artilcles~~.xml.bz2</code> (This is the latest revision of every Wikipedia page, article and template - the ~~basis~~ you need to get going)		#* <code>enwiki-(date)-pages-articles-multistream.xml.bz2</code> (This is the latest revision of every Wikipedia page, article and template - the basics you need to get going)
	#* <code>enwiki-~~<date>-redirect~~.~~sql.bz2</code> (This will make redirects function correctly)~~		# This is compressed with Bzip2 - if you have the space, extract it once downloaded to speed up importing.
	~~#* <code>enwiki-<date>-templatelinks.sql.bz2</code> (This will make the template links function correctly)~~
	~~#* <code>enwiki-<date>-site_stats.sql.bz2</code> (This will fill in article counts~~ and ~~the like without searching for you)~~		=== Downloading and installing MediaWiki ===
	~~# Put all the files in a dedicated folder so they are all available in one place for later.~~

	~~=== Downloading MWDumper ===~~		* Download the latest version of Mediawiki from the [https://www.mediawiki.org Mediawiki] website. As were using Linux, it's better to download the .tar.gz version.
			* Extract the archive
			* Clear out anything not needed:
			** Timeless and Mono skin
			** Text files in the root, install.sh, docker...
			* Copy the folder to the webroot
			* Place a file in the .../resources/asset folder if using a picture for the site logo/favicon

	~~# <s>MWDumper is available from many places around~~ the Internet, both in source form and already built Java packages. You will need to download a copy from [https://integration.wikimedia.org/ci/view/Java/job/MWDumper-package/ Jenkins], this is pre-built by MediaWiki. MWDumper 1.16 (26/06/2013) was the latest at the time of writing.		=== Preparing the database ===
	# You will need to remove any versions of OpenJDK already installed (remove <code>libreoffice-calc-extensions</code> and <code>libreoffice-writer-extensions</code> before OpenJDK so that it doesn't try to install another version of Java).
	~~# You will then need to install the latest Oracle Java JRE and JDK packages (64-bit packages are safe and better for this as we don't need the web plugin)</s>~~

	=== ~~Downloading and installing MediaWiki~~ ===		* Login to MariaDB as root
			* Create the database:
			CREATE DATABASE enwiki;
			* Create a user for the database:
			CREATE USER 'enwiki'@'localhost' IDENTIFIED BY 'database_password';
			A password can be generated at [https://passwordsgenerator.net/ Password Generator Plus]. Use a length as long as possible, it doesn't need to be remembered past this configuration.
			* Grant priveliges for the user to this database:
			GRANT ALL PRIVILEGES ON enwiki.* TO 'enwiki'@'localhost' WITH GRANT OPTION;
			* Exit MariaDB
			* Restart the server
			systemctl restart mariadb

			This can be tweaked with different database and user names as required.

			=== Moving databse to a different hard drive ===

			Due to the sheer size of the databse, you may choose to move the MariaDB database to a different drive. MariaDB stores each database in a seperate folder by default making this easy.

			* Stop MariaDB
			systemctl stop mariadb
			* Navigate to <code>/var/lib/mysql</code>
			* Move the <code>enwiki</code>/database name folder to where you want the database to be stored
			* Back in the <code>/var/lib/mysql</code> folder, create a symlink to where you moved the folder, using the same name for the symlink
			* Chown the databse folder where you moved it to mysql:
			chown -R mysql:root /path/to/folder/enwiki
			* Restart MariaDB, check it starts with no errors
			systemctl start mariadb

			=== Install Mediawiki ===

			* Navigate to where your instance is installed: for example, https://enwiki.freddythechick.net/. You will be greeted by the Mediawiki installer.

			== Importing the dump ==