A follow-on to the risk catalogue from previous assignment and DW post. Again here is a messy blog post on methodology and what the cat was doing (yowling) while download LibO to my iMac and as I struggle to put together the formal report specified in the assignment.
LibO lastName entry points
I started off this morning at the VW dealership with some exploratory testing to see if any of the spare few test ideas I have noted are feasible/sensible. I quickly realized a few key things. To keep my test case collection manageable I should probably settle on one or at most two ways to input values of the variable under test. In my initial risk catalog write-up (cf prev post) I identified three major entry points: start-up wizard, options, read from user settings.( goes on for quite a bit, it does )
Some notes on tools
While blundering around I found many better ways to do what I was up to. Even without a formal harness it's easy to generate and evaluate data against a SUT this way with the built-ins of any scripting language. I settled on Python because after trying some Ruby and Perl I got going the fastest in Python .. though as I found easily enough I was still going much slower than I needed to. I should have googled sooner as that would have saved me some time and keystrokes.
Here are two things I will use in the future for this sort of exploration: Python libraries to interact directly with the platform clipboard and (if available) the Ruby black bag security toolset. There's no coincidence at all that the same tools that work for neutral/academic software testing and QA are also used to attack software and find its weak point and I've studied both.
Ed note: I've submitted the assignment to the class and will post it here after the course is done and grading is completed.
The core attributes for function testing are coverage, validity, and affordability. Desired characteristics of a function test (plan) are performability and reuseability. Function testing considers whether individual functions (features, groups, of functions, etc) are performing correctly and function test plans are usually based on covering all of the functions adequately with tests. At such a small scale validity is very important for function tests or they have little value. Function testing can be automated which may reduce costs. Automated function testing can be implemented into development procedures (as unit testing often is) which allows for reduced cost through reuse. The test suite can assert that the operation fo the functions tested is unchanged after changes to the SUT by developers.
The core attributes for risk based test design are credibility, information value, motivation, and power and a desirable characteristic of risk test is affordability. Risk based testing seeks to find potential failure modes fo the SUT, produce them, and gather information about their likelihood and consequences. Good risk based test design should find demonstrable bugs which case noteworthy failures in the SUT including failures that may affect other systems. These tests should help convince stakeholders to make needed changes based on the potential risks identified and demonstrated. Risk based testing can vary in it's application between quicktests which are effective for certain common failure types to more complex and expensive failure mode analysis such as that performed by engineers in safety industries.
Core attributes of specification testing are accountability, coverage, and value. Often specification testing is for purposes of determining compliance with formal specification which may have legal weight. In these cases the details fo testing may be published or even be part of the public record through a standards body or court proceeding. In these cases coverage of the entire formal specification by the test plan is completely required and the information gathered is of considerable importance to the organization running the tests. Other kinds of specification testing, such as competitive product analysis, also provide good value to the testing organization.
- Share presentations with your friends and coworkers. Upload and convert existing presentations to Google Docs format.
- Download your presentations as a PDF, a PPT, or a .txt file.
- Insert images and videos, and format your slides.
- Publish and embed your presentations in a website, allowing access to a wide audience.
- Draw organizational charts, flowcharts, design diagrams and much more right within a presentation.
- Add slide transitions, animations, and themes to create show-stopping presentations.
- See exactly what others are working on with colorful presence markers
- Edit a presentation with other people simultaneously from different locations
- Use revision history to see who made changes or to revert to earlier versions
- Say hello, start a conversation or share new ideas using built-in chat
- Create Google documents, spreadsheets, other file types, and collections.
- Upload (from your computer, if you'd like), manage, and store files and folders.
- Share Google Docs, files, and collections.
- Preview your docs and files before you open or share them.
- View images and videos that you've uploaded to your Documents List.
- Search for items by name, type, and visibility setting.
- Convert most file types to Google Docs format.
- Add flair and format your documents, with options such as paint format, margins, spacing, and fonts.
- Invite other people to collaborate on a doc with you, giving them edit, comment or view access.
- Collaborate online in real time and chat with other collaborators.
- View your documents' revision history and roll back to any version.
- Download Google Docs to your desktop as Word, OpenOffice, RTF, PDF, HTML or zip files.
- Translate a document to a different language.
- Email your documents to other people as attachments.
- Share and edit presentations with your friends and coworkers.
- Import and convert existing presentations in .ppt and .pps file types.
- Download your presentations as a PDF, a PPT, or a .txt file.
- Insert images and videos, and format your slides.
- Allow real-time viewing of presentations, online, from separate remote locations.
- Publish and embed your presentations in a website, allowing access to a wide audience.
- Share and edit drawings with your friends and coworkers.
- Download your presentations as a PNG, JPEG, SVG, or PDF file.
- Insert images, shapes, and lines, and format them to fit your preferences.
- Real-time collaboration with other people, no matter where they are.
- Insert a drawing into a document, spreadsheet, or presentation.
- Presentations is an online application and runs in a web browser and on Google servers.
- Presentation requires Internet access to Google servers and other Internet resources for full use of features.
- Presentations accepts file upload from the web browser in certain file formats and outputs files.
- Presentation files may also be embedded into other web sites.
- Presentations uses Google's shared user authentication and authorization systems and Google search.
- Presentation is free to use without charge for anyone with a Google account.
- Structure: and Operations: Presentations is delivered as a online service (SaaS) and there is no physical product.
- Platform: Google's platform is used for (at least) storage, authn, authz, search, revision control, machine translation, document format conversions, and application delivery.
- Data: Presentations reads in and writes out in multiple well-documented formats and also has an internal format.
- Compatibility: Presentations reads and writes multiple file formats besides it's native data format.
- Compatibility: Presentations is sensitive to web browser feature levels.
- Installability criteria may not be applicable since Presentation is delivered as a service
- Security in Presentations is implemented with features of Google hosting services and may not be directly testable.
- Capability: A serious failure or lack of any of the features bragged about in the manual will strongly impact quality
To help us learn how to digest and actually get useful information from specs and other complex documents the lectures and reading explain active reading techniques and emphasize the use of mind-mapping software. The assignment is to use a mind mapping application to make a map of a specification and answer some questions about the results. If you haven't got or aren't familiar with the mind mapping tool you are encouraged to snag it and start in early in the assigned time for this assignment , and I did yesterday with mixed results.
Here's where it gets choppy: I haven't met a mind mapping program that I can actually use effectively, though I have tried a few a few times. Much as no note-taking application is faster or more versatile for me than scribbles on paper (alas I would this were not true, see recent /. discussion for ample discourse. tl;dr use a pen and paper) I have to whiteboard or pen sketch flowcharts, timelines, swimlanes and especially formal maps (at work) before trying to fight them into a computer. So this assignment's technique is unlikely to work well for me however awesome it is. And confirmation bias as it may, I had enough trouble inputting the skeleton into the mapper yesterday that I'm pretty convinced it's a net-loss for efficiency and don't want to use it again, certainly not for inputting data.
But the real problem is that the specification document we are supposed to analyze isn't a spec. It's the bleedin' online manual and is mostly full of marketing and fluff. I haven't seen any numbers in the parts I have tried to skim and if there's a section of fluff about interoperability I haven't been able to find it yet.
I think I could have surmounted (kludged) one of these two problems but with both staring me down I'm locking up. I should be able to analyze this app from the fluffy manual and using it, but I won't have anyhting in 3-4 hours but a headache and a wall-white board full of scribble which it would take me a couple hours to clean up and get into the mapper (Or faster into Visio, Omnigraffle, Inkscape in order of speed and cost).
I've either got to learn to type quickly or the computers need to learn to understand my scribble and/or when I yell at them .. but this shouldn't have anything to do with how to analyze a spec or mock the thing they gave you in the specification folder.
I guess I should try and active read through the spec, taking notes as best I can (still no study skills to speak of) and ignore the map for the remaining 2-3 hours and try and come up with something. It either that or I'll drawn a moderately useful map and need another couple hours to get it into the shiny metal box on my table here. Argh.
ETA: a snip from the assignment to demonstrate the gulf between these techniques and anything that will actually work for me:
Every sentence of a specification should be telling you what the product is (Product Elements), in what way it is good or bad or needs to get better (Quality Criteria), or how it will be built and the context in which it will be built (Project Environment). As you find information about the product, note it under one of the topics or subtopics under these main headings.
This sounds like something best accomplished with printouts, scissors, and maybe a bunch of index cards. Then once you have something maybe you can put it in a computer. Am I really so far out on this? How can anyone actually organize a bunch of random crap on a computer?
ETA: Pics or it didn't ...
1) The variable of lastname is input (optionally) in the wizard that runs on first program execution of OOo applications, modified in the Options dialogue of any OOo application, and used in every facet of the office suite.
2a) Undefined is a valid state for this variable and any code path that uses the variable without checking for undef or doing so incorrectly will introduce errors in its functions. Additionally, too much data in this field would also be dangerous to any of the code that uses it do the likelihood of buffer overruns and unexpected characters or encoding in this field could lead to format string errors or exceptions in library string-handling code. If you can get a non-character or non-string value into this field due to input validation failures then wholesale memory and stack corruption becomes a concern.
2b) Use of lastname as well as the companion variables first name and initials is widespread throughout the SUT applications. Beyond the dialogs which directly manipulate this value (new user wizard, Options) many other functions read this variable and incorporate it into interface displays (document properties) or include it in requests to other modules (printing). The name variables are included in various places in the document data saved to disk automatically and intentionally including the document properties. If change tracking is enabled a tag generated from name variables and dates is displayed next to each change made by a particular user and recored with version information in the document files. Perhaps most excitingly the name variables are posted to Internet servers with registration information allowing for the small possibility that an error related to this variable could affect not only systems that process the document but completely remote systems as well!
2bi) Lastname is used in (at least) many display functions in all parts of the SUT applications, change tracking functions, save/load functions, printing functions, macros, user preferences, data generating functions such as headers and footers, and online registration.
2bii) Values of lastname are displayed in numerous parts of the UI, in change-tracking feature's tags, inside saved documents (and temp/autosaved ones), and may be printed depending on settings for header/footer and cover pages.
2biii) Values of lastname are sent to the operating system as part of stored data about the user and document as well as to remote devices for printing (settings dependent) and to remote Internet servers with optional software registration. I'm unsure about how lastname values may be used in API calls and macros.
2biv) Values of lastname are sent to the operating system as part of stored dat
a about the user (registry UserProfile.xcu) and document (meta.xml).
2bv) Values are read from the registry user profile files if available and may be input into the SUT via the first-run wizard or on demand with Options dialogs.
2c) Changes to the presence or boundedness (?) of last name during program operations could lead to corrupted data in memory, on disk, and displayed to users.
2cviii) Display of user data, document data or metadata could be impacted by incorrect information about the presence (undef), values (could change), or boundedness (wrong data type) of last name.
2cix) Boundary errors on lastname could influence other variable values in document metadata, document content, or application configuration leading to problems with these unrelated variables and functions. Gross misbehaviour on the part of lastname could completely corrupt XML program and document data structures rendering the document or preferences unreadable and thus broadly disrupting document or application functionality.
2cx) There any number of cases where software errors could cause the value of lastname currently in memory and on disk in the user registry or document to become de-synched. This could lead to incorrect data being saved or printed. Some of these cases include local or remote file system errors, unaccounted for 'races' with other OOo (or alien) process running on either the SUT or the file storage device, or just faulty RAM.
2cxi) Lastname is optionally sent with registration information to remote Internet servers operated by the OOo project. A chained failure of input filtering or other unlikely occurrence could cause unexpected format, encoding, or sized lastname to be injected into the remote system and processed. It is not entirely far-fetched that this could lead to serious problems on the remote system(s) that receive and process the data such as a buffer overflow or SQL injection attack.
2cxii) Perhaps the most outlandish and unlikely risk is to the hardware of a printer that receives a document to print with malformed or corrupt lastname information in the cover page, headers, or document body. The could lead the printer to malfunction, develop sentience, or start making toast if enough failures chain together in just the right (wrong?) way.
( Risks for OOo Last Name: a continuum of failure stretching towards disaster )
I am quite excited (and nervous) to be diving into a new BBST course after how much I learned from the previous ones (and how much work they were).
I'm a security analyst in a small business unit of a really large company and before that I was in IT as a system administrator and what all else.
Cheerfully enough I live within walking distance of the office and try to take advantage of that as often as weather permits.
I use testing techniques, especially those learned in BBST courses, in a lot of non-development software work including vulnerability assessment and configuration management. I have a sneaking suspicion there is a crossover between security analysis and software testing methodologies in my future careers ...
I like testing and automation tools because they support my scientific focus on dealing with computers and users both. I demand reproducibility and hooks for automation in, well, practically everything ... and not just at work. I've been heard to cry out "It's computer science, not computer superstition!" on occasion when reboots(!) are suggested as a solution** to a problem.
Outside of work or software even I read escapist fiction and nerdy non-fiction, watch some telly, bemoan how I'm not keeping up with my foreign language studies, crafting, or martial arts lessons, and then get distracted by video games, cats, or other humans.
** Workaround they may be but not a solution and they destroy any hope of researching that instance of the problem...
Still, I am troubled by these conflicts and felt I should write up what's going on. I'm not happy to be scraping the course rules and I'm really concerned about this making the final still more difficult than it was going to already be.
Today is already better. Up at the luxurious weekend hour of 8:30 I have had tea and checked email. I stretched out a bit and did 15 minutes on the fake bike. A paltry sum to be sure, but an important psychological victory to complete a short "ride" without getting a cramp or otherwise injuring myself :/ And the CSI I put on to watch while "riding" isn't too bad. Second cup of tea is steeping.
I misunderstood the course schedule of assignments in my favour. This whole week I have merely to answer and discuss sample exam questions in the forums and try to grade two other students Phase 3 writings (as well as two more videos and quizzes). The final will be next week, which is a much better scheme for me being able to finish this week before the con. It's still plenty of work but not nearly as bad as I had thought.
I think roomie and her boyfriend are here somewhere ... haven't seen them really since dinner Thursday. I'm waiting until a decent hour to go take a shower, say 11 ?
Must be a feature?
( shell transcript )
* read over the final exam study guide, (terrifying)
* read over the assignment guidelines (phase 1 is due at midnight tonight latest),
* read over a dozen OOo Impress bugs, (I need to comment on one)
* installed OOo on the Dell mini 12 (might be able to reproduce a crash bug on that platform!)
* grazed on crackers, crisps and jerky
* finished a cup of citrusy green and brewing a cup of PGtips now.
* made a presentation of no worth in OOo Impress to get familiar with it.
I still have to:
* pick a bug report, and then both comment usefully on it and write about the report
* make intelligent sounding remarks on answers to previous exercise (others and my own, should have been done earlier in the week)
And this is just the first week of the crazy accelerated 4 week class. WTF was I thinking trying this again while working fulltime?
** So far: Mommy's Little Monster, Social Distortion, Prison Bound
Oracle OrientationSuppose you were writing a text editing program that would display fonts properly on the screen. How would you test whether the font display is correct?
- How could you determine whether the editor is displaying text in the correct typeface?
- How could you determine whether the editor is displaying text in the right size?
- How would you test whether the editor displays text correctly (in any font that the user specifies)? Describe your thinking about this problem and the strategies you are considering.
I'm [adric]. I'm a System Administrator for a web hosting company on weekend nights, doing some contract work for a small professional education organization some weekdays, and manage to spend a few minutes each week trying to help the OLPC* folks save the world**, play some video games, and study a bit. I live in one or more suburbs of Atlanta, GA, USA, grew up in another suburb, and work downtown and in the suburbs, variously.
I'm fascinated by and often chasing technology in many of it's varied forms.
In my tiny amount of software development work I first keyed onto basic testing tools (continuous building, unit testing) as part of project automation and quickly became enamoured of the tools which enable me to write better code. I also came into an interest in testing through reading and study computer security. The importance of developer testing, QA, and live evaluation is driven home by the literature (e.g. comp.risks) in security fields over and over again. More broadly I see testing technology in many forms as a way to try and move software forward as a technology so computers may become more reliable and useful to everyone.
In the past week I played some Dungeons and Dragons with friends, finished the PC game Mass Effect, started it again, finished re-reading the Dune novels (Frank Herbert), fooled around with a Japanese childrens' game on Nintendo DS (すもくかわいね), and hassled both of the house cats. I watch movies and some television, although mostly on my Mac laptop.
* The One Laptop per Child (OLPC) foundation aims to put educational tools into the hands of all children worldwide. More at http://laptop.org/
** Saving the world is progressing, but they may have failed to save themselves... heavy staff cuts this week.
I am quite sure I have a lot to learn from this course and all of you.
Thanks for reading,
attached: A picture from the NOC here at work from Jan 2007