Viin SEM - Crawler
by Viindoo
| Lines of Code | 1707 |
| Technical name | viin_sem_crawler |
| License | OPL-1 |
| Website | https://viindoo.com/apps/modules/17.0/viin_sem_crawler |
| Read description for | v 15.0 v 16.0 v 18.0 |
| Required Apps | Discuss (mail) Website (website) |
| Included Dependencies | Viindoo Base Viin SEM Viin Lang Detection |
| Extensions | SEM Crawler and Helpdesk Integrator |
Crawl your website: status codes, links, images, changes - and a blacklist
Each website URL is fetched (no redirects followed), its content, text, type and response time stored as a crawl, links and images parsed with BeautifulSoup and checked, child URLs created, changes detected, broken children counted; a cron crawls periodically, sitemap URLs are generated, old crawls removed, and a blacklist keeps login or private pages out.
Viin SEM - Crawler (technical name viin_sem_crawler) is an Odoo 17.0 app for Odoo Community and Odoo Enterprise and Viindoo Cloud, built for seo specialist, webmaster.
At a Glance
The facts of Viin SEM - Crawler, version 0.1.4, in one place. Published by Viindoo.
How It Works
Enable automatic crawl on the website or press Crawl on a URL; review crawls and invalid children.
A URL
Status 200, crawls, response time.
Links
Internal, external, status.
Crawls
Date, status, links, images.
A crawl
Content type, text.
Blacklist
Login and portal home.
URLs
200 and 404 side by side.
Settings
Automatic crawl, retention.
What You Get
Fetch with status, type, response time
Links and images parsed and checked
Child URLs created
Change detection between crawls
Blacklist with reason
Automatic crawl and sitemap cron
Crawl retention per website
What This App Does Not Do
Read this before you buy. Everything below is something the app deliberately leaves to another app or to you.
A 3xx is recorded as such.
Others are stored.
It must reach the website.
Works Well With
Apps from the same stack, built to fit this one:
Who Should Use Viin SEM - Crawler?
SEO specialist
/en/shop answers 404 - seen the same day.
Webmaster
Login pages blacklisted, never crawled.
Frequently Asked Questions
What does Viin SEM - Crawler do?
Each website URL is fetched (no redirects followed), its content, text, type and response time stored as a crawl, links and images parsed with BeautifulSoup and checked, child URLs created, changes detected, broken children counted; a cron crawls periodically, sitemap URLs are generated, old crawls removed, and a blacklist keeps login or private pages out.
What does Viin SEM - Crawler not do?
Redirects are not followed. A 3xx is recorded as such. Only text and image content types are parsed. Others are stored. Crawling runs from the Odoo server. It must reach the website.
Who is Viin SEM - Crawler for?
SEO specialist: /en/shop answers 404 - seen the same day. Webmaster: Login pages blacklisted, never crawled.
Which Odoo version and editions does it support?
Odoo 17.0 - Odoo Community, Odoo Enterprise, Viindoo Cloud. Upgrades to a newer Odoo series are a separate purchase for that series.
What does it depend on?
It installs on top of: viin_sem. Odoo installs them with it.
How do I set it up?
Enable automatic crawl on the website or press Crawl on a URL; review crawls and invalid children.
What works well with it?
Viin SEM (viin_sem): Required. SEM Helpdesk (viin_sem_helpdesk): Tickets from broken links.
Can I try it before buying?
Yes - the Live Preview button at the top of this page opens the module's own screen on a working database.
How do I get support?
Write to apps.support@viindoo.com with your Odoo version and the technical name viin_sem_crawler; pre-sales questions go to sales@viindoo.com.
See Viin SEM - Crawler in Action
Live demo: v17demo-int.viindoo.com/web#action=viin_sem.website_url_action
Need help with Viin SEM - Crawler?
For questions, implementation support or a custom feature, contact Viindoo.
Pre-Sales & Partnership
sales@viindoo.comUpgrades to a newer Odoo series are a separate purchase for that series.
All Viindoo apps: apps.odoo.com/apps/modules/browse?author=Viindoo
Technical Requirement
Changes log
0.1.4 - Latest on the 17.0 line
- Aligned with the 17.0 models and views.
This software and associated files (the "Software") may only be used (executed, modified, executed after modifications) if you have purchased a valid license from the authors, typically via Odoo Apps, or if you have received a written agreement from the authors of the Software (see the COPYRIGHT file).
You may develop Odoo modules that use the Software as a library (typically by depending on it, importing it and using its resources), but without copying any source code or material from the Software. You may distribute those modules under the license of your choice, provided that this license is compatible with the terms of the Odoo Proprietary License (For example: LGPL, MIT, or proprietary licenses similar to this one).
It is forbidden to publish, distribute, sublicense, or sell copies of the Software or modified copies of the Software.
The above copyright notice and this permission notice must be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.