-
-
-
+
+
+
+
+
Community
+ +Community is what really sets ArduPilot apart from many other offerings in the + market. Here are some of the places you can find ArduPilot users and developers:
+ + +Latest posts on Community Blogs
++ +
Latest Community Forums posts
+-
+
+ + +
Discussion Forum
++ + + + + + +
+
Facebook ArduPilot group
+Discord
++ + + + + +
+ + +
Developer Community
++ Working on the code? You can start with our Developers + Guide and chat with other developers at our discord + channels +
+-
-
Community
-Community is what really sets ArduPilot apart from many other offerings in the - market. Here are some of the places you can find ArduPilot users and developers:
+What is ArduPilot?
+ArduPilot is a trusted, versatile, and open source autopilot system supporting many + vehicle types: multi-copters, traditional helicopters, fixed wing aircraft, boats, submarines, rovers and + more. + The source code is developed by a large community of professionals and enthusiasts. New developers are + always + welcome! The best way to start is by joining the Developer Team Forum, which is open to all and chock-full + of + daily development goodness.
++
One autopilot for any mission
+ArduPilot enables the creation and use of trusted, autonomous, unmanned vehicle + systems for the peaceful benefit of all. ArduPilot provides a comprehensive suite of tools suitable for + almost + any vehicle and application. As an open source project, it is constantly evolving based on rapid feedback + from a + large community of users. The Development Team works with the community and commercial partners to add + functionality to ArduPilot thatbenefits everyone. Although ArduPilot does not manufacture any hardware, + ArduPilot firmware works on a wide variety of different hardware to control unmanned vehicles of all types. + Coupled with ground control software, unmanned vehicles running ArduPilot can have advanced functionality + including real-time communication with operators. ArduPilot has a huge online community dedicated to helping + users with questions, problems, and solutions.
+ +The ArduPilot Project provides an advanced, full-featured and reliable open source + autopilot software system. The first ArduPilot open code repository was created in 2009 - since then it has + been + developed by a team of diverse professional engineers, academics, computer scientists, and other members of + our + global community. It is capable of controlling almost any vehicle system imaginable: conventional and VTOL + airplanes, gliders, multirotors, helicopters, sailboats, powered boats, submarines, ground vehicles and even + Balance-Bots. The supported vehicle types frequently expand as use cases emerge for new and novel + platforms.
+ +Installed in over 1,000,000 vehicles world-wide, and with advanced data-logging, + analysis and simulation tools, ArduPilot is a deeply tested and trusted autopilot system. The open-source + code + base means that it is rapidly evolving, always at the cutting edge of technology development, whilst sound + release processes provide confidence to the end user. With many peripheral suppliers creating interfaces, + users + benefit from a broad ecosystem of sensors, companion computers and communication systems. Since the source + code + is open, it can be audited to ensure compliance with security and secrecy requirements.
+ +The software suite is installed in vehicles from many manufacturers, such as many + from our Partners, and more broadly throughout the global autonomous systems industry. It is also used for + testing and development by large institutions and corporations such as NASA, Intel and Insitu/Boeing, as + well as + countless colleges and universities around the world.
+-
Latest posts on Community Blogs
-+
-
Latest Community Forums posts
--
-
+
Terms & Slices
+-
Discussion Forum
+Unmanned vehicles
++
- - - - - +
+
Hardware
++
Facebook ArduPilot group
+Firmware
++ + + +
Software
+Discord
+Ground Station
+- - - - +
Mission Planner
++ + +
- - -
Developer Community
-- Working on the code? You can start with our Developers - Guide and chat with other developers at our discord channels -
-- -
- - -
What is ArduPilot?
-ArduPilot is a trusted, versatile, and open source autopilot system supporting many - vehicle types: multi-copters, traditional helicopters, fixed wing aircraft, boats, submarines, rovers and more. - The source code is developed by a large community of professionals and enthusiasts. New developers are always - welcome! The best way to start is by joining the Developer Team Forum, which is open to all and chock-full of - daily development goodness.
--
One autopilot for any mission
-ArduPilot enables the creation and use of trusted, autonomous, unmanned vehicle - systems for the peaceful benefit of all. ArduPilot provides a comprehensive suite of tools suitable for almost - any vehicle and application. As an open source project, it is constantly evolving based on rapid feedback from a - large community of users. The Development Team works with the community and commercial partners to add - functionality to ArduPilot thatbenefits everyone. Although ArduPilot does not manufacture any hardware, - ArduPilot firmware works on a wide variety of different hardware to control unmanned vehicles of all types. - Coupled with ground control software, unmanned vehicles running ArduPilot can have advanced functionality - including real-time communication with operators. ArduPilot has a huge online community dedicated to helping - users with questions, problems, and solutions.
- -The ArduPilot Project provides an advanced, full-featured and reliable open source - autopilot software system. The first ArduPilot open code repository was created in 2009 - since then it has been - developed by a team of diverse professional engineers, academics, computer scientists, and other members of our - global community. It is capable of controlling almost any vehicle system imaginable: conventional and VTOL - airplanes, gliders, multirotors, helicopters, sailboats, powered boats, submarines, ground vehicles and even - Balance-Bots. The supported vehicle types frequently expand as use cases emerge for new and novel platforms.
- -Installed in over 1,000,000 vehicles world-wide, and with advanced data-logging, - analysis and simulation tools, ArduPilot is a deeply tested and trusted autopilot system. The open-source code - base means that it is rapidly evolving, always at the cutting edge of technology development, whilst sound - release processes provide confidence to the end user. With many peripheral suppliers creating interfaces, users - benefit from a broad ecosystem of sensors, companion computers and communication systems. Since the source code - is open, it can be audited to ensure compliance with security and secrecy requirements.
- -The software suite is installed in vehicles from many manufacturers, such as many - from our Partners, and more broadly throughout the global autonomous systems industry. It is also used for - testing and development by large institutions and corporations such as NASA, Intel and Insitu/Boeing, as well as - countless colleges and universities around the world.
+ArduPilot Documentation
+ArduPilot has a large set of documentation available through its Wiki. See the link + DOCS in the top bar to check it. You need to choose your hardware (which you can get help in "AutoPilot + Hardware"), then you could follow vehicle's "First Time Setup" for installing the firmware and Ground + Station + software, connecting the components, and calibration steps.
-
+
Terms & Slices
-- -
Unmanned vehicles
--
Hardware
--
Firmware
-- - - -
Software
-Ground Station
-User Cases
Mission Planner
-- - -
- -
ArduPilot Documentation
-ArduPilot has a large set of documentation available through its Wiki. See the link - DOCS in the top bar to check it. You need to choose your hardware (which you can get help in "AutoPilot - Hardware"), then you could follow vehicle's "First Time Setup" for installing the firmware and Ground Station - software, connecting the components, and calibration steps.
-- - - - +
+
User Cases
-+
+
-
- - - - - + }); + }) + .catch(error => console.error('Error:', error)); + + + diff --git a/frontend/js/carousel_data.js b/frontend/js/carousel_data.js new file mode 100644 index 0000000000..bca7b02bdf --- /dev/null +++ b/frontend/js/carousel_data.js @@ -0,0 +1,177 @@ +let carouselData = [ + { + "imgSrcWebp": "images/avoidance.webp", + "imgSrcJpeg": "images/avoidance.jpg", + "imgAlt": "Avoidance example" + }, + { + "imgSrcWebp": "images/slides_02_7_hour_flight_Titled_920.webp", + "imgSrcJpeg": "images/slides_02_7_hour_flight_Titled_920.jpg", + "imgAlt": "Airbus VTOL UAV" + }, + { + "imgSrcWebp": "images/slides_03_Airbus-local-motors.webp", + "imgSrcJpeg": "images/slides_03_Airbus-local-motors.jpg", + "imgAlt": "Airbus-local-motors" + }, + { + "imgSrcWebp": "images/slides_3b_2048x1536-1.webp", + "imgSrcJpeg": "images/slides_3b_2048x1536-1.jpg", + "imgAlt": "3b_2048x1536" + }, + { + "imgSrcWebp": "images/slides_04_AP_Discourse_AntiMatterCrusaderBoat_.webp", + "imgSrcJpeg": "images/slides_04_AP_Discourse_AntiMatterCrusaderBoat_.jpg", + "imgAlt": "AP_Discourse_AntiMatterCrusaderBoat" + }, + { + "imgSrcWebp": "images/slides_05_Rubidium_Rover.webp", + "imgSrcJpeg": "images/slides_05_Rubidium_Rover.jpg", + "imgAlt": "slides_05_Rubidium_Rover" + }, + { + "imgSrcWebp": "images/slides_06_Olivier_Brousse_Methane_Copter.webp", + "imgSrcJpeg": "images/slides_06_Olivier_Brousse_Methane_Copter.jpg", + "imgAlt": "slides_06_Olivier_Brousse_Methane_Copter" + }, + { + "imgSrcWebp": "images/slides_07_zhangir.webp", + "imgSrcJpeg": "images/slides_07_zhangir.jpg", + "imgAlt": "slides_07_zhangir" + }, + { + "imgSrcWebp": "images/slides_09_luiz_goncalves.webp", + "imgSrcJpeg": "images/slides_09_luiz_goncalves.jpg", + "imgAlt": "slides_09_luiz_goncalves" + }, + { + "imgSrcWebp": "images/slides_10_vlcsnap-2017-02-21-.webp", + "imgSrcJpeg": "images/slides_10_vlcsnap-2017-02-21-.jpg", + "imgAlt": "slides_10_vlcsnap-2017-02-21-" + }, + { + "imgSrcWebp": "images/slides_11_Aerial_Alchemy.webp", + "imgSrcJpeg": "images/slides_11_Aerial_Alchemy.jpg", + "imgAlt": "slides_11_Aerial_Alchemy" + }, + { + "imgSrcWebp": "images/slides_12_Facebook_Adam_hubalovsky.webp", + "imgSrcJpeg": "images/slides_12_Facebook_Adam_hubalovsky.jpg", + "imgAlt": "slides_12_Facebook_Adam_hubalovsky" + }, + { + "imgSrcWebp": "images/slides_13_OBC16Caneberra1.webp", + "imgSrcJpeg": "images/slides_13_OBC16Caneberra1.jpg", + "imgAlt": "slides_13_OBC16Caneberra1" + }, + { + "imgSrcWebp": "images/slides_14_OBC16Canberra3.webp", + "imgSrcJpeg": "images/slides_14_OBC16Canberra3.jpg", + "imgAlt": "slides_14_OBC16Canberra3" + }, + { + "imgSrcWebp": "images/slides_15_OBC16Canberra2.webp", + "imgSrcJpeg": "images/slides_15_OBC16Canberra2.jpg", + "imgAlt": "slides_15_OBC16Canberra2" + }, + { + "imgSrcWebp": "images/slides_16_Volcano1.webp", + "imgSrcJpeg": "images/slides_16_Volcano1.jpg", + "imgAlt": "slides_16_Volcano1" + }, + { + "imgSrcWebp": "images/slides_17_HYBRiX-flying-landinggear.webp", + "imgSrcJpeg": "images/slides_17_HYBRiX-flying-landinggear.jpg", + "imgAlt": "slides_17_HYBRiX-flying-landinggear" + }, + { + "imgSrcWebp": "images/slides_18_BlueRov-920.webp", + "imgSrcJpeg": "images/slides_18_BlueRov-920.jpg", + "imgAlt": "slides_18_BlueRov-920" + }, + { + "imgSrcWebp": "images/slides_19_UAV3000_1500.webp", + "imgSrcJpeg": "images/slides_19_UAV3000_1500.jpg", + "imgAlt": "slides_19_UAV3000_1500" + }, + { + "imgSrcWebp": "images/slides_20_RLNovaerial_square.webp", + "imgSrcJpeg": "images/slides_20_RLNovaerial_square.jpg", + "imgAlt": "slides_20_RLNovaerial_square" + }, + { + "imgSrcWebp": "images/slides_21_NovAerial_GCS1.webp", + "imgSrcJpeg": "images/slides_21_NovAerial_GCS1.jpg", + "imgAlt": "slides_21_NovAerial_GCS1" + }, + { + "imgSrcWebp": "images/slides_22_NovAerialGCS2.webp", + "imgSrcJpeg": "images/slides_22_NovAerialGCS2.jpg", + "imgAlt": "slides_22_NovAerialGCS2" + }, + { + "imgSrcWebp": "images/slides_23_3DVistas.webp", + "imgSrcJpeg": "images/slides_23_3DVistas.jpg", + "imgAlt": "slides_23_3DVistas" + }, + { + "imgSrcWebp": "images/slides_24_MIT_Andrew_Barry_1920.webp", + "imgSrcJpeg": "images/slides_24_MIT_Andrew_Barry_1920.jpg", + "imgAlt": "slides_24_MIT_Andrew_Barry_1920" + }, + { + "imgSrcWebp": "images/slides_25_cat_eddie_weeks_2-920.webp", + "imgSrcJpeg": "images/slides_25_cat_eddie_weeks_2-920.jpg", + "imgAlt": "slides_25_cat_eddie_weeks_2" + }, + { + "imgSrcWebp": "images/slides_26_cat_eddie_weeks_mission_resize.webp", + "imgSrcJpeg": "images/slides_26_cat_eddie_weeks_mission_resize.jpg", + "imgAlt": "slides_26_cat_eddie_weeks_mission_resize" + }, { + "imgSrcWebp": "images/slides_27_home_ardupilot.webp", + "imgSrcJpeg": "images/slides_27_home_ardupilot.jpg", + "imgAlt": "slides_27_home_ardupilot" + }, + { + "imgSrcWebp": "images/slides_28_Greg_Covey_FireFLY6.webp", + "imgSrcJpeg": "images/slides_28_Greg_Covey_FireFLY6.jpg", + "imgAlt": "slides_28_Greg_Covey_FireFLY6" + }, + { + "imgSrcWebp": "images/slides_29_bg2.webp", + "imgSrcJpeg": "images/slides_29_bg2.jpg", + "imgAlt": "slides_29_bg2" + }, + { + "imgSrcWebp": "images/slides_30_octaquadplane.webp", + "imgSrcJpeg": "images/slides_30_octaquadplane.jpg", + "imgAlt": "slides_30_octaquadplane" + }, + { + "imgSrcWebp": "images/slides_31_SkyMapper_1152px.webp", + "imgSrcJpeg": "images/slides_31_SkyMapper_1152px.jpg", + "imgAlt": "slides_31_SkyMapper_1152px" + }, + { + "imgSrcWebp": "images/slides_32_vlcsnap-2017-02-21-15h48m21s139-920.webp", + "imgSrcJpeg": "images/slides_32_vlcsnap-2017-02-21-15h48m21s139-920.jpg", + "imgAlt": "slides_32_vlcsnap-2017-02-21-15h48m21s139-920" + }, + { + "imgSrcWebp": "images/slides_33_Thomas_Coyle_Rover_Check.webp", + "imgSrcJpeg": "images/slides_33_Thomas_Coyle_Rover_Check.jpg", + "imgAlt": "slides_33_Thomas_Coyle_Rover_Check" + }, + { + "imgSrcWebp": "images/slides_34_zion1413273025362.webp", + "imgSrcJpeg": "images/slides_34_zion1413273025362.jpg", + "imgAlt": "slides_34_zion1413273025362" + }, + { + "imgSrcWebp": "images/slides_35_BlueRov-920.webp", + "imgSrcJpeg": "images/slides_35_BlueRov-920.jpg", + "imgAlt": "slides_35_BlueRov-920" + }, + // continue listing all carousel items... +]; diff --git a/frontend/scripts/get_discourse_posts.py b/frontend/scripts/get_discourse_posts.py index 405f476d4f..c370a6bc90 100755 --- a/frontend/scripts/get_discourse_posts.py +++ b/frontend/scripts/get_discourse_posts.py @@ -46,10 +46,11 @@ def __init__(self, blog_url: str, news_url: str): @staticmethod def get_arguments() -> Any: - parser = argparse.ArgumentParser(description="python3 get_discourse_posts.py [Number of posts to retrieve]") + parser = argparse.ArgumentParser(description="python3 get_discourse_posts.py") parser.add_argument("--n_posts", dest='n_posts', default="8", help="Number of posts to retrieve") parser.add_argument("--verbose", dest='verbose', action='store_false', default=True, help="show debugging output") - return parser.parse_args() + args, unknown = parser.parse_known_args() + return args @staticmethod def execute_http_request_json(url: str) -> Any: @@ -87,10 +88,14 @@ def get_first_youtube_link(request: str) -> str: # Regular expression to find URLs that contain 'YouTube' or image links url_pattern = re.compile(r'href=[\'"]?(https?://www\.youtube[^\'" >]+)') img_pattern = re.compile(r'(?:href|src)=[\'"]?(https?://[^\'" >]+\.(jpg|jpeg|png|gif|svg|bmp|webp))') + img_pattern2 = re.compile(r'img src=[\'"]?(https?://[^\'" >]+)') # catch google link and such # Find all matches youtube_links = url_pattern.findall(first_five_lines) img_links = img_pattern.findall(first_five_lines_lower)[0] if img_pattern.findall(first_five_lines_lower) else None + if img_links is None: + img_links = img_pattern2.findall(first_five_lines_lower)[0] if img_pattern2.findall( + first_five_lines_lower) else None # If there are image links before YouTube links, return empty string if img_links and (not youtube_links or diff --git a/js/jquery-3.2.1.min.js b/js/jquery-3.2.1.min.js deleted file mode 100644 index 644d35e274..0000000000 --- a/js/jquery-3.2.1.min.js +++ /dev/null @@ -1,4 +0,0 @@ -/*! jQuery v3.2.1 | (c) JS Foundation and other contributors | jquery.org/license */ -!function(a,b){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){"use strict";var c=[],d=a.document,e=Object.getPrototypeOf,f=c.slice,g=c.concat,h=c.push,i=c.indexOf,j={},k=j.toString,l=j.hasOwnProperty,m=l.toString,n=m.call(Object),o={};function p(a,b){b=b||d;var c=b.createElement("script");c.text=a,b.head.appendChild(c).parentNode.removeChild(c)}var q="3.2.1",r=function(a,b){return new r.fn.init(a,b)},s=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,t=/^-ms-/,u=/-([a-z])/g,v=function(a,b){return b.toUpperCase()};r.fn=r.prototype={jquery:q,constructor:r,length:0,toArray:function(){return f.call(this)},get:function(a){return null==a?f.call(this):a<0?this[a+this.length]:this[a]},pushStack:function(a){var b=r.merge(this.constructor(),a);return b.prevObject=this,b},each:function(a){return r.each(this,a)},map:function(a){return this.pushStack(r.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(f.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(a<0?b:0);return this.pushStack(c>=0&&c0&&b-1 in a)}var x=function(a){var b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u="sizzle"+1*new Date,v=a.document,w=0,x=0,y=ha(),z=ha(),A=ha(),B=function(a,b){return a===b&&(l=!0),0},C={}.hasOwnProperty,D=[],E=D.pop,F=D.push,G=D.push,H=D.slice,I=function(a,b){for(var c=0,d=a.length;c
You can change and check the parameters for another version: + +
\ No newline at end of file + + document.getElementById('selectPicker').addEventListener('change', function() { + window.location.href = this.value; + }); + + function appendToSelect(json) { + var selectPicker = document.getElementById('selectPicker'); + for (var key in json) { + var opt = document.createElement('option'); + opt.value = json[key]; + opt.innerHTML = key; + selectPicker.appendChild(opt); + } + } +}); + diff --git a/planner/source/docs/mission-planner-overview.rst b/planner/source/docs/mission-planner-overview.rst index ec97b49226..a0e1da575c 100644 --- a/planner/source/docs/mission-planner-overview.rst +++ b/planner/source/docs/mission-planner-overview.rst @@ -31,7 +31,7 @@ Planner: - With appropriate telemetry hardware you can: - Monitor your vehicle's status while in operation. - - Record telemetry logs which contain much more information the the + - Record telemetry logs which contain much more information about the on-board autopilot logs. - View and analyze the telemetry logs. - Operate your vehicle in FPV (first person view) diff --git a/rover/source/_static/parameters_versioning_script.inc b/rover/source/_static/parameters_versioning_script.inc index 8dc5fe548c..7ca4146daa 100644 --- a/rover/source/_static/parameters_versioning_script.inc +++ b/rover/source/_static/parameters_versioning_script.inc @@ -1,11 +1,27 @@ -You can change and check the parameters for another version: - - +
You can change and check the parameters for another version: + +
\ No newline at end of file + + document.getElementById('selectPicker').addEventListener('change', function() { + window.location.href = this.value; + }); + + function appendToSelect(json) { + var selectPicker = document.getElementById('selectPicker'); + for (var key in json) { + var opt = document.createElement('option'); + opt.value = json[key]; + opt.innerHTML = key; + selectPicker.appendChild(opt); + } + } +}); + diff --git a/update.py b/update.py index 8fd4d8638e..7ca9062706 100755 --- a/update.py +++ b/update.py @@ -25,7 +25,7 @@ using the "site" shortcode: [site wiki="plane,rover"]conditional content[/site] -Parameters files are fetched from autotest using a Wget +Parameters files are fetched from autotest using requests """ from __future__ import print_function, unicode_literals @@ -46,6 +46,11 @@ import subprocess import sys import time +import requests +from urllib.parse import urlparse +from concurrent.futures import ThreadPoolExecutor +from typing import Optional, Dict, List + import rst_table @@ -55,6 +60,8 @@ # without the following import on old versions of Python: from distutils import dir_util # noqa: F401 +from frontend.scripts import get_discourse_posts + if sys.version_info < (3, 8): print("Minimum python version is 3.8") sys.exit(1) @@ -108,7 +115,7 @@ def debug(str_to_print): """Debug output if verbose is set.""" if VERBOSE: - print("[update.py] " + str_to_print) + print(f"[update.py]: {str_to_print}") def error(str_to_print): @@ -132,69 +139,103 @@ def remove_if_exists(filepath): raise e -def fetch_url(fetchurl): - '''fetches content at url and puts it in a file corresponding to the filename in the URL''' - if platform.system() == "Windows": - subprocess.check_call(["powershell.exe", "Start-BitsTransfer", "-Source", fetchurl]) - else: - subprocess.check_call(["wget", fetchurl]) +def fetch_and_rename(fetchurl: str, target_file: str, new_name: str) -> None: + fetch_url(fetchurl, fpath=new_name, verbose=False) + # move in new file + if os.path.exists(target_file): + os.unlink(target_file) + os.rename(new_name, target_file) -def fetchparameters(site=None, cache=None): - """Fetches the parameters for all the sites from the test server and - copies them to the correct location. +def fetch_url(fetchurl: str, fpath: Optional[str] = None, verbose: bool = True) -> None: + """Fetches content at url and puts it in a file corresponding to the filename in the URL""" + print(f"Fetching {fetchurl}") - This is always run as part of a build (i.e. no checking to see if - parameters have changed.) + if verbose: + total_size = get_request_file_size(fetchurl) - """ - # remove any parameters files in root - remove_if_exists('Parameters.rst') + response = requests.get(fetchurl, stream=True) + response.raise_for_status() - for key, value in PARAMETER_SITE.items(): - fetchurl = 'https://autotest.ardupilot.org/Parameters/%s/Parameters.rst' % value - targetfile = './%s/source/docs/parameters.rst' % key - if key == 'AP_Periph': - targetfile = './dev/source/docs/AP_Periph-Parameters.rst' - if cache: - if not os.path.exists(targetfile): - raise Exception("Asked to use cached parameter files, but (%s) does not exist" % (targetfile,)) - continue - if site == key or site is None: - fetch_url(fetchurl) + filename = fpath or os.path.basename(urlparse(fetchurl).path) - # move in new file - if os.path.exists(targetfile): - os.unlink(targetfile) - os.rename('Parameters.rst', targetfile) + downloaded_size = 0 + chunk_size = 10 * 1024 + with open(filename, 'wb') as out_file: + if verbose: + print(f"Completed : 0%", end='') + completed_last = 0 + for chunk in response.iter_content(chunk_size=chunk_size): + out_file.write(chunk) + downloaded_size += len(chunk) -def fetchlogmessages(site=None, cache=None): - """ - Fetches the parameters for all the sites from the autotest server and + # progress bar + if verbose: + completed = downloaded_size * 100 // total_size + if completed - completed_last > 10 or completed == 100: + print(f"..{completed}%", end='') + completed_last = completed + if verbose: + print() # Newline to correct the console cursor position + + +def get_request_file_size(url: str) -> int: + headers = {'Accept-Encoding': 'identity'} # needed as request use compression by default + hresponse = requests.head(url, headers=headers) + + if 'Content-Length' in hresponse.headers: + size = int(hresponse.headers['Content-Length']) + return size + return 0 + + +def fetchparameters(site: Optional[str] = None, cache: Optional[str] = None) -> None: + dataname = "Parameters" + fetch_ardupilot_generated_data(PARAMETER_SITE, f'https://autotest.ardupilot.org/{dataname}', f'{dataname}.rst', + f'{dataname.lower()}.rst', site, cache) + + +def fetchlogmessages(site: Optional[str] = None, cache: Optional[str] = None) -> None: + dataname = "LogMessages" + fetch_ardupilot_generated_data(LOGMESSAGE_SITE, f'https://autotest.ardupilot.org/{dataname}', f'{dataname}.rst', + f'{dataname.lower()}.rst', site, cache) + + +def fetch_ardupilot_generated_data(site_mapping: Dict, base_url: str, sub_url: str, document_name: str, + site: Optional[str] = None, cache: Optional[str] = None) -> None: + """Fetches the data for all the sites from the test server and copies them to the correct location. This is always run as part of a build (i.e. no checking to see if - logmessages have changed.) + parameters or logmessage have changed.) + """ - for key, value in LOGMESSAGE_SITE.items(): - fetchurl = 'https://autotest.ardupilot.org/LogMessages/%s/LogMessages.rst' % value - targetfile = './%s/source/docs/logmessages.rst' % key + urls: List[str] = [] + targetfiles: List[str] = [] + names: List[str] = [] + + for key, value in site_mapping.items(): + fetchurl = f'{base_url}/{value}/{sub_url}' + targetfile = f'./{key}/source/docs/{document_name}' + if key == 'AP_Periph': + targetfile = f'./dev/source/docs/AP_Periph-{sub_url}' if cache: if not os.path.exists(targetfile): - raise Exception("Asked to use cached parameter files, but (%s) does not exist" % (targetfile,)) + raise Exception(f"Asked to use cached files, but {targetfile} does not exist") continue if site == key or site is None: - fetch_url(fetchurl) - # move in new file - if os.path.exists(targetfile): - os.unlink(targetfile) - os.rename('LogMessages.rst', targetfile) + urls.append(fetchurl) + targetfiles.append(targetfile) + names.append(f"{value}_{document_name}") + + with ThreadPoolExecutor() as executor: + executor.map(fetch_and_rename, urls, targetfiles, names, timeout=5*60) def build_one(wiki, fast): '''build one wiki''' - debug('Using make for sphinx: %s' % wiki) + print(f'Using make for sphinx: {wiki}') if platform.system() == "Windows": # This will fail if there's no folder to clean, so no check_call here if not fast: @@ -219,7 +260,7 @@ def sphinx_make(site, parallel, fast): done.add(wiki) if site == 'common' or site == 'frontend': continue - if wiki == 'frontend' : + if wiki == 'frontend': continue if site is not None and not site == wiki: continue @@ -295,13 +336,12 @@ def copy_build(site, destdir): shutil.move(sourcedir, html_moved_dir) # Rename move! (single move to html/* failed) shutil.move(html_moved_dir, targetdir) - debug("Moved to %s" % targetdir) - except Exception: # FIXME: narrow exception type - error("FAIL moving output to %s" % targetdir) + debug(f"Moved to {targetdir}") + except shutil.Error: + error(f"FAIL moving output to {targetdir}") # copy jquery os.makedirs(os.path.join(targetdir, '_static'), exist_ok=True) - shutil.copy(os.path.join('js', 'jquery-3.2.1.min.js'), os.path.join(targetdir, '_static', 'jquery-3.2.1.min.js')) # delete the old directory debug('Removing %s' % olddir) @@ -357,6 +397,13 @@ def delete_old_wiki_backups(folder, n_to_keep): error('Error on deleting some previous wiki backup folders: %s' % e) +def create_dir_if_not_exists(dir_path: str) -> None: + try: + os.mkdir(dir_path) + except FileExistsError: # Catching specific exception + pass + + def generate_copy_dict(start_dir=COMMON_DIR): """ This creates a dict which indexes copy targets for all common docs. @@ -374,25 +421,10 @@ def generate_copy_dict(start_dir=COMMON_DIR): # Create destination folders that might be needed (if don't exist) for wiki in ALL_WIKIS: - try: - os.mkdir(wiki) - except Exception: # FIXME: narrow exception type - pass - - try: - os.mkdir('%s/source' % wiki) - except Exception: # FIXME: narrow exception type - pass - - try: - os.mkdir('%s/source/docs' % wiki) - except Exception: # FIXME: narrow exception type - pass - - try: - os.mkdir('%s/source/_static' % wiki) - except Exception: # FIXME: narrow exception type - pass + create_dir_if_not_exists(wiki) + create_dir_if_not_exists(f'{wiki}/source') + create_dir_if_not_exists(f'{wiki}/source/docs') + create_dir_if_not_exists(f'{wiki}/source/_static') for root, dirs, files in os.walk(start_dir): for file in files: @@ -487,46 +519,11 @@ def fix_site_shortcode(matchobj): def logmatch_code(matchobj, prefix): - try: - print("%s m0: %s" % (prefix, matchobj.group(0))) - except Exception: # FIXME: narrow exception type - print("%s: except m0" % prefix) - - try: - print("%s m1: %s" % (prefix, matchobj.group(1))) - except Exception: # FIXME: narrow exception type - print("%s: except m1" % prefix) - - try: - print("%s m2: %s" % (prefix, matchobj.group(2))) - except Exception: # FIXME: narrow exception type - print("%s: except m1" % prefix) - - try: - print("%s m3: %s" % (prefix, matchobj.group(3))) - except Exception: # FIXME: narrow exception type - print("%s: except m3" % prefix) - - try: - print("%s m4: %s" % (prefix, matchobj.group(4))) - except Exception: # FIXME: narrow exception type - print("%s: except m4" % prefix) - try: - print("%s m5: %s" % (prefix, matchobj.group(5))) - except Exception: # FIXME: narrow exception type - print("%s: except m5" % prefix) - try: - print("%s m6: %s" % (prefix, matchobj.group(6))) - except Exception: # FIXME: narrow exception type - print("%s: except m6" % prefix) - try: - print("%s m7: %s" % (prefix, matchobj.group(7))) - except Exception: # FIXME: narrow exception type - print("%s: except 7" % prefix) - try: - print("%s m8: %s" % (prefix, matchobj.group(8))) - except Exception: # FIXME: narrow exception type - print("%s: except m8" % prefix) + for i in range(9): + try: + print("%s m%d: %s" % (prefix, i, matchobj.group(i))) + except IndexError: # The object has less groups than expected + print("%s: except m%d" % (prefix, i)) def is_the_same_file(file1, file2): @@ -555,9 +552,8 @@ def fetch_versioned_parameters(site=None): if key == 'AP_Periph': # workaround until create a versioning for AP_Periph in firmware server fetchurl = 'https://autotest.ardupilot.org/Parameters/%s/Parameters.rst' % value - subprocess.check_call(["wget", fetchurl]) targetfile = './dev/source/docs/AP_Periph-Parameters.rst' - os.rename('Parameters.rst', targetfile) + fetch_and_rename(fetchurl, targetfile, 'Parameters.rst') else: # regular versining @@ -740,16 +736,13 @@ def put_cached_parameters_files_in_sites(site=None): pass -def update_frotend_json(): +def update_frontend_json(): """ Frontend get posts from Forum server and insert it into JSON """ debug('Running script to get last posts from forum server.') try: - if platform.system() == "Windows": - subprocess.check_call(["python", "./frontend/scripts/get_discourse_posts.py"]) - else: - subprocess.check_call(["python3", "./frontend/scripts/get_discourse_posts.py"]) + get_discourse_posts.main() except Exception as e: error(e) pass @@ -761,7 +754,7 @@ def copy_static_html_sites(site, destdir): """ if (site in ['frontend', None]) and (destdir is not None): debug('Copying static sites (only frontend so far).') - update_frotend_json() + update_frontend_json() folder = 'frontend' try: site_folder = os.getcwd() + "/" + folder @@ -777,7 +770,7 @@ def check_imports(): '''check key imports work''' import pkg_resources # package names to check the versions of. Note that these can be different than the string used to import the package - requires = ["sphinx_rtd_theme>=1.0.0", "sphinxcontrib.youtube>=1.2.0", "sphinx==5.1.1", "docutils==0.16"] + requires = ["sphinx_rtd_theme>=1.3.0", "sphinxcontrib.youtube>=1.2.0", "sphinx>=7.1.2", "docutils<0.19"] for r in requires: debug("Checking for %s" % r) try: